Imagine you had bought this newspaper for a friend. Imagine you asked them to tell you what's in the TV listings - and they demanded cash before they would tell you. Outrageous? Certainly. Yet that is what a number of government agencies are doing with the data that we, as taxpayers, pay to have collected on our behalf. You have to pay to get a useful version of that data. Think of Ordnance Survey's (OS) mapping data: useful to any business that wanted to provide a service in the UK, yet out of reach of startup companies without deep pockets.
This situation prevails across a number of government agencies. Its effects are all bad. It stifles innovation, enterprise and the creativity that should be the lifeblood of new business. And that is why Guardian Technology today launches a campaign - Free Our Data. The aim is simple: to persuade the government to abandon copyright on essential national data, making it freely available to anyone, while keeping the crucial task of collecting that data in the hands of taxpayer-funded agencies.
One government makes the data it collects available free to all: the United States. It is no accident that it is also the country that has seen the rise of multiple mapping services (such as Google Maps, Microsoft's MapPoint and Yahoo Maps) and other services - "mashups" - that mesh government-generated data with information created by the companies. The US takes the attitude that data collected using taxpayers' money should be provided to taxpayers free. And a detailed study shows that the UK's closed attitude to its data means we lose out on commercial opportunities, and even hold back scientific research in fields such as climate change.
Who are the culprits? Besides OS, with its vast and valuable map data, there are the UK Hydrographic Office (which collects tidal and naval navigational data), the Highways Agency (which collects traffic data) and even the European Centre for Medium Range Weather Forecasting.
Britain's public sector information is held by some 400 government departments, agencies and local authorities. Assets range from wills dating back to 1858, house values recorded in the Land Registry, maps and the risk of flooding to individual homes. Much is of great commercial interest, especially when it can be presented on innovative websites such as upmystreet.com. These sets of data are the modern crown jewels - but instead of treating them as a resource to boost national wealth, the government locks them up, restricting access to those who pay.
The heart of our argument is twofold. First, the government should not run businesses. The civil service is too inflexible to cope with the speed of change in the commercial sector. If a company like Microsoft finds it difficult to adjust to a world of online applications, how much harder is it for a government department rooted in paper-based processes?
Second, the government should be charged with collecting the best data. The Office for National Statistics labours to collect the most accurate depiction of Britain's society and economy. No private organisation has the resources, time or breadth of approach. By contrast, commercial companies ignore less profitable sectors: we can't trust data collected by banks about spending habits because they ignore people who cannot afford to open accounts. Similarly, few companies make detailed maps of Britain; they concentrate on the areas with the best returns.
In a seminal piece of research into the real cost of charging for access to public data, the late Peter Weiss, of the US National Weather Service, compared open and closed economic models for public sector data. His paper, Borders in Cyberspace: Conflicting Public Sector Information Policies and their Economic Impact, is online (http://tinyurl.com/cby55). He quoted a 2000 study for the European Commission carried out by Pira International, which noted that "the concept of commercial companies being able to acquire, at very low cost, quantities of public sector information and resell it for a variety of unregulated purposes to make a profit is one that policymakers in the EU find uncomfortable." But why?
Pira pointed out that the US's approach brings enormous economic benefits. The US and EU are comparable in size and population; but while the EU spent €9.5bn (£6.51bn) on gathering public sector data, and collected €68bn selling and licensing it, the US spent €19bn - twice as much - and realised €750bn - over 10 times more. Weiss pointed out: "Governments realise two kinds of financial gain when they drop charges: higher indirect tax revenue from higher sales of the products that incorporate the ... information; and higher income tax revenue and lower social welfare payments from net gains in employment."
Infinite applications
The applications for public sector data are infinite, but here are two real-world ones that affect ordinary businesses. Many of Britain's best rock-climbing venues are on sea cliffs, and hence affected by the tides. For climbers planning a trip - and surely spending money in local shops - it helps to know if the tides will be favourable. But websites that try to offer British tide data have been told by the UK Hydrographic Office they must pay for it - a cost most are unwilling to endure. So sites have no tides, climbers make the safer choice, and shops miss out.
Take traffic data. The Highways Agency has set up an exclusive deal with TrafficTV to send video details from motorway cameras to your mobile. But if the data were free, any company could offer well-organised maps and drive down the price while increasing the numbers who would have better access to traffic data. That would reduce congestion and pollution.
How did we get into this mess? The problem stems from the days of the Conservative government, which created "trading funds". These government-owned bodies - including OS, Companies House, the Driving Standards Agency, HM Land Registry, the Patent Office and the Royal Mint - have to sell to other parts of the government and private customers. Every year they have a "revenue target" that must be paid to the Treasury; but the only way to earn those revenues is through commercial activity - creating the ludicrous spectacle of government-approved monopolies making the market buy data at their rates. A growing number of experts say the loss to the economy outweighs "gains" through lowered taxes. "The amount of money made is trivial," says Keith Dugmore of the Demographics User Group, which represents commercial users of government information.
But the stifling of competition is most serious. Sometimes, the conflicts are surreal. The supply of postal addresses is an example. Three publicly owned agencies - the Post Office, OS and a consortium of local authorities - run lists of addresses. Because they are forced to put a "value" on the lists, they have been unable to agree a way of creating a single database. The cost of licensing addresses was one reason why returns from the 2001 census were so poor from several fast-changing metropolitan areas, such as Manchester.
A local authority such as Swindon has to pay OS £38,000 a year to use its addresses and geographical data. It also has to pay the Royal Mail £3,000 for every website that includes the facility for people to look up their postcodes. Yet it was local authorities, which have a statutory duty to collect street addresses, that collected much of this data.
Intense conflict
Kristin Woodland, who chairs the local authorities' street gazetteer group, says: "The taxpayer pays for us to create the data, then has to pay us to use the data."
The conflict between a trading fund's traditional role and its new commercial activities is most intense in mapping. "Technology has left the licensing model far behind," says Dugmore.
Particular controversy surrounds the role of OS, founded to prepare the British army for a French invasion and now a household name that sees itself at the forefront of geographical information systems.
OS has a monopoly on centuries' worth of data thanks to taxpayer funding. Without that head start, it would be in the same position as any other startup today: facing the choice of creating a brand-new dataset, or finding someone who had collected it and licensing it from them.
Part of the problem is that government - and the Treasury - is blinded by its internal commercial targets, and unable to see the damage its policies do. Thus within government, OS is widely seen as an example of a venerable public body transforming itself into a dynamic commercial enterprise. Last year it had a surplus of £9.2m, though only part was returned to the Treasury. But the bulk of its revenues come from the public sector, partly through a payment called the National Interest Mapping Services (Nims), worth £13m a year, and partly from deals with government bodies.
The latest of the deals is under negotiation. Yesterday, the government was due to receive bids for a £20m contract to provide geographical information to Whitehall bodies. In theory, it is an open competition. In practice, the largest part of the contract, for "large scale data", is unlikely to go to anyone else: the contract specifies that the winner must have large-scale maps in place within six weeks of the start of a five-year deal.
The OS told the Guardian: "We are bidding for pan-government agreement and we operate in fair competition under heavy scrutiny. The only monopoly we have is in the mapping of uneconomic areas." In 2004-05, 47.5% of its £100.4m trading revenue came from government (which allows it to claim that it is not government funded). It says Nims is not a subsidy: "This is a contribution to the cost of mapping uneconomic areas."
Profit centres
That makes sense - if you consider the geography of the United Kingdom to be a profit centre. But if you view it instead as belonging to its taxpayers, and meriting rigorous mapping for their benefit, there are no "uneconomic areas" - only places that people haven't started to use yet.
Happily, the practice of state-owned monopolies competing in markets dependent on their information is under attack from several quarters. A new trade association, Locus, is calling for the government to enforce a level playing field in the market in public sector information.
And some of the most copyright-mad government organisations have relented. Hansard, the record of proceedings in parliament, was once a paper product that had to be bought from Her Majesty's Stationery Office. Now it is online, for free, so that sites like theyworkforyou.com can generate useful data about individual MPs' voting and attendance.
The Office of Fair Trading is preparing a report on public sector information, due this summer, which will "look at whether or not the way in which public sector information holders (PSIH) supply information works well for businesses. It will examine whether PSIHs have an unfair advantage selling on information in competition with companies who are reliant on the PSIH for that raw data in the first place."
Though it may already be shooting for an open goal, we urge the OFT to compare the UK with the US; read Peter Weiss's paper; and then, finally, to free our data.
· If you'd like to comment on any aspect of Technology Guardian, send your emails to tech@theguardian.com