Location Cube: 2013

Mapstraction is a Javascript library that provides a single, common interface for a wide variety of Javascript map APIs. It’s designed to enable a developer to switch from one maps API to another as smoothly and as quickly as possible.

http://davidrowley.blogspot.co.nz/2013/08/calculating-line-of-sight-with-postgis.html

Performing Line of Sight analysis from PostGIS

http://www.imagico.de/

Map generalisation (coastlines) and ray tracing

http://www.klokantech.com/maptiler/

Powerful technology for accelerated pre-rendering of large maps

http://mapproxy.org/

MapProxy is an open source proxy for geospatial data. It caches, accelerates and transforms data from existing map services and serves any desktop or web GIS client.

https://github.com/NYPL/map-vectorizer

Vectorize raster maps including OCR of text

This project aims to automate a manual process: geographic polygon and attribute data extraction from maps including those from insurance atlases published in the 19th and early 20th centuries.

Tuesday, July 23, 2013

PowerShell–batch search/replace in filenames

I needed to rename a set of files , keeping the original name but adding a suffix. Powershell can do this quite easily using the following:

PS C:\data\photos> Dir | Rename-Item -NewName { $_.name -replace ".jpg","_840.jpg" }

this will take all .jpg files in the folder and change the name to include an _840.jpg… which in my case was the vertical dimensions of the image set.

Thursday, July 11, 2013

Powershell – how to renumber files sequentially

Sourced from http://justanothersysadmin.wordpress.com/2008/03/22/bulk-rename-files-with-sequential-index/

here is how to use powershell to renumber files sequentially.. be careful not to renumber files you don’t want renumbered!! Best to move all the files to a new temp folder to do this….

1) copy paste as single line into powershell

function Rename-Bulk($prefix){$files = Get-ChildItem;$id = 1;$files | foreach { Rename-Item -Path $_.fullname -NewName ( $prefix + ((($id++).tostring()).padleft(($files.count.tostring()).length) -replace ' ','0' ) + $_.extension) }}

2) CD to folder in powershell

3) Rename-Bulk('img') where img is the prefix for all output with sequential number following

Powershell – Merging files to then load into PostgreSQL

Merging a set of files into one large file

Start PowerShell then use CD to move to the Directory with the files to merge

Get-Content .\*.txt | Out-File c:\temp\merged.txt

This will find all .txt (text) files and merge into a single document in c:\temp folder.

However I found the output was in Unicode format… and to then load this into PostgreSQL
So I then loaded and saved the file using textpad in UTF-8 format
then create the table in PG….

create table houseprice03_msoa_ew
(la_code text,
la_name text,
msoa_code text,
msoa_name text,
price_all_dwellings_2percentile integer,
price_all_dwellings_lower_quartile integer,
price_all_dwellings__median integer,
price_all_dwellings__upper_quartile integer,
price_all_dwellings_98precentilentile integer,
price_all_dwellings__mean integer
)

and then copy the data into that table…

COPY houseprice03_msoa_ew FROM 'C:\Data\GIS\MartynDorey\CensusData\2003 Houseprices\houseprice2003_G080309_2288_GeoPolicy_MSOA.csv' DELIMITER ',' CSV HEADER;

OSM in 3D – F4

I read a Tweet about a new 3D visualisation of OSM using a new rendering engine.

It includes some animation (eg water surfaces, and fountains)…

Overall the graphic is pretty good.. but the actual data is still poor.

It assumes a flat world with no terrain, and building heights are somewhat arbitrary.

Here’s a view of Edinburgh (Scotland).. looking along Princes Street with Edinburgh Castle on the right, and Calton Hill in the distance. In reality the Castle is way up high on a volcanic plug (hill)… and the gardens infront of it are in a valley.

http://map.f4-group.com/#lon=-3.2010387&lat=55.9519892&zoom=17&camera.theta=79.141&camera.phi=-69.901

Here’s another view…. this time looking at the Castle from above East Princes Street Gardens (near The Mound)

Here’s a quick comparison of the first image from F4 and Google Earth… as you can see GE includes the topography.

Wednesday, June 26, 2013

Installing an ASPX app to IIS on Win2008r2

Steps to install a web app on IIS

1) Write ASPx code etc in Vis Studio

2) Make a folder somewhere and in VS choose publish to File- putting in that folder

3)Zip up that folder and copy folder to server somewhere

4) For some reason the install application button is missing on IIS…

Follow this to add the button:

http://randypaulo.wordpress.com/2011/08/22/iis-7-5-missing-import-application-deployment/

Which tells you to install web deploy from MS

http://www.iis.net/downloads/microsoft/web-deploy

5) Then IMPORT the application – set to the default web site

6) Right click on the folder you imported in IIS and CONVERT TO APPLICATION

7) Possible issues:

- you might need to make a new application pool (check left side of IIS management console for Application Pools) – then set the application to an application pool which is integrated and using .net 4+

- You might need to enable 32bit applications for that new application pool – advanced properties and set 32bit apps to TRUE

- you might find ASPX.NET service isn’t running.. check SERVICES from Start menu / control panel – and set from manual to automatic.. and start it running

- you might need to compile your code for ANY CPU

- don’t forget to copy across all .dlls should happen automatically but worth checking they’re all in the bin folder as required

Tuesday, May 14, 2013

PostGIS WITH RECURSIVE

I was wondering how to walk networks without using pgRouting…. and found Paul Ramsey’s blog had the answer…

The text here is taken from his blog.. backed up here for future quick reference.

http://blog.cleverelephant.ca/2010/07/network-walking-in-postgis.html

CREATE TABLE network ( segment geometry, id integer primary key );

INSERT INTO network VALUES ('LINESTRING(1 1, 0 0)', 1);
INSERT INTO network VALUES ('LINESTRING(2 1, 1 1)', 2);
INSERT INTO network VALUES ('LINESTRING(1 2, 1 1)', 3);
INSERT INTO network VALUES ('LINESTRING(3 1, 2 1)', 4);
INSERT INTO network VALUES ('LINESTRING(3 2, 2 1)', 5);
INSERT INTO network VALUES ('LINESTRING(2 3, 1 2)', 6);
INSERT INTO network VALUES ('LINESTRING(1 3, 1 2)', 7);
INSERT INTO network VALUES ('LINESTRING(4 2, 3 2)', 8);
INSERT INTO network VALUES ('LINESTRING(3 4, 2 3)', 9);
INSERT INTO network VALUES ('LINESTRING(2 4, 2 3)', 10);
INSERT INTO network VALUES ('LINESTRING(1 4, 1 3)', 11);
INSERT INTO network VALUES ('LINESTRING(4 3, 4 2)', 12);
INSERT INTO network VALUES ('LINESTRING(4 4, 3 4)', 13);

CREATE INDEX network_gix ON network USING GIST (segment);

WITH RECURSIVE walk_network(id, segment) AS (
    SELECT id, segment FROM network WHERE id = 6
  UNION ALL
    SELECT n.id, n.segment
    FROM network n, walk_network w
    WHERE ST_DWithin(ST_EndPoint(w.segment),ST_StartPoint(n.segment),0.01)
  )
SELECT id
FROM walk_network

Which returns:

 id
----
  6
  3
  1
(3 rows)

This can be made into a function like this:

CREATE OR REPLACE FUNCTION downstream(integer)
RETURNS geometry
LANGUAGE sql
AS '
WITH RECURSIVE walk_network(id, segment) AS (
    SELECT id, segment FROM network WHERE id = $1
  UNION ALL
    SELECT n.id, n.segment
    FROM network n, walk_network w
    WHERE ST_DWithin(ST_EndPoint(w.segment),ST_StartPoint(n.segment),0.01)
  )
SELECT ST_MakeLine(ST_EndPoint(segment))
FROM walk_network
' IMMUTABLE;

And here’s the function in action, generating the downstream path from segment 12.

=# SELECT ST_AsText(Downstream(12));

            st_astext
---------------------------------
 LINESTRING(4 2,3 2,2 1,1 1,0 0)
(1 row)

Monday, April 22, 2013

Crowd Sourced Information– Some Sources

Here are some useful links for accessing Crowd Sourced data:

Twitter

https://github.com/Twitterizer/Twitterizer

FlickR

http://flickrnet.codeplex.com/

Bike Data for Shared Bike Schemes:

http://www.citybik.es/ - check the API pages

> Example of its use: http://bikes.oobrien.com/london/

Tuesday, March 12, 2013

Useful things to know when using SQLite and .NET

This is worth knowing when using SQLite and the System.Data.Sqlite .Net library.

1) Various connection strings

Source: http://www.connectionstrings.com/sqlite

e.g.

using cache> Data Source=c:\mydb.db;Version=3;Cache Size=3000;

using a memory database> Data Source=:memory:;Version=3;New=True;

read only> Data Source=c:\mydb.db;Version=3;Read Only=True;

Journal files>
Data Source=c:\mydb.db;Version=3;Journal Mode=Off;
Data Source=c:\mydb.db;Version=3;Synchronous=Full;
(means full flush after each write)

Using compression >
Data Source=c:\mydb.db;Version=3;Compress=True;

2) DB locking issues and how to get around it

Source: http://www.lastaddress.net/2010/11/systemdatasqlite-locks-database-even-on.html

using (SqliteConnection sqlCon = new SqliteConnection(connStr))
{

sqlCon.Open();
using(SqliteCommand sqlCom = new SqliteCommand(sqlCon, sqlStr))

{

     using(SqliteDataReader rdr = sqlCom.ExecuteReader())

     {

        //do data stuff

        rdr.Close();

     }

     sqlCon.Close(); //CLOSE sqlCon BEFORE sqlCom IS DESTROYED!

}
}

Sunday, January 20, 2013

Central Logger and Dashboard tool –Web App

I’ve been working the last 2 weeks on a central logging tool, and a web dashboard. The idea is that any component in a system can post a log record to the central logging server.

Log sentences are defined to be 4 parts long… these are message type (msgtype), message part 1 (msg1), message part 2 (msg2), and message part 3 (msg3). Each part is sent as a string. The server runs a concurrent memory queue, each TCP client sends a log… and the server time and task id are added at the time they arrive.

Every so often (eg 5 seconds or 100 new entries) the memory queue is cleared in a transactional update to the central SQL Lite DB. From testing it seems that this easily supports 10 concurrent users posting 10,000 records at the rate of 2000 inserts per second across the internet. Which is fine for what we need.

The second part is the dashboard, which I built using ASPX and a web service. It also uses OpenLayers to connect to a GeoServer instance to display map content. I use Flexigrid (http://flexigrid.info/) to display the attribute data and allow for sorting and filtering.

I had 3 main issues with taking the running application from the dev environment to operational. These were:

1) the SQL Lite library supports 32bit and 64bit. I’d dev on a 32bit machine but was trying to run it on a 64bit server. I got around this by creating an IIS application pool and setting it to allow 32bit apps, then assigning this to the web app I’m deploying.

2) The service wasn’t available from any remote web browser, it only ran on the local machine (the web server itself). To fix this I had to add some extra lines to the webconfig…

inside of the <configuration> <system.web>
add this section

For more info: http://support.microsoft.com/kb/819267

3) Everything was running fine on the IIS server…until after a while the locationservice.svc was reported to be not be available from the browser.. the error the server reported was:

“Could not load file or assembly App_Web...”

So I copied the files across again to the IIS server and it would all be fine for a while again… until an hour or so later packing up again. This turned out to be an issue where the server recompiles the ASPX code every so often. To turn this facility off you need to add the following to the web config… under <system.web>

For more on this error and the fix check this link:

http://stackoverflow.com/questions/4051576/could-not-load-app-web-dll-exception-after-asp-net-compile-recycle

Location Cube

Search This Blog

Friday, December 13, 2013

Photosynth Tech Preview

Tuesday, December 10, 2013

Python and GIS + BING live data feed for travel

Web Links Dec 2013

Wednesday, September 4, 2013

Sept 2013–List of Interesting Map Web tech