Search This Blog

Tuesday, May 14, 2013

PostGIS WITH RECURSIVE

I was wondering how to walk networks without using pgRouting…. and found Paul Ramsey’s blog had the answer… 


The text here is taken from his blog.. backed up here for future quick reference.


http://blog.cleverelephant.ca/2010/07/network-walking-in-postgis.html




CREATE TABLE network ( segment geometry, id integer primary key );

INSERT INTO network VALUES ('LINESTRING(1 1, 0 0)', 1);
INSERT INTO network VALUES ('LINESTRING(2 1, 1 1)', 2);
INSERT INTO network VALUES ('LINESTRING(1 2, 1 1)', 3);
INSERT INTO network VALUES ('LINESTRING(3 1, 2 1)', 4);
INSERT INTO network VALUES ('LINESTRING(3 2, 2 1)', 5);
INSERT INTO network VALUES ('LINESTRING(2 3, 1 2)', 6);
INSERT INTO network VALUES ('LINESTRING(1 3, 1 2)', 7);
INSERT INTO network VALUES ('LINESTRING(4 2, 3 2)', 8);
INSERT INTO network VALUES ('LINESTRING(3 4, 2 3)', 9);
INSERT INTO network VALUES ('LINESTRING(2 4, 2 3)', 10);
INSERT INTO network VALUES ('LINESTRING(1 4, 1 3)', 11);
INSERT INTO network VALUES ('LINESTRING(4 3, 4 2)', 12);
INSERT INTO network VALUES ('LINESTRING(4 4, 3 4)', 13);

CREATE INDEX network_gix ON network USING GIST (segment);



Network


 




WITH RECURSIVE walk_network(id, segment) AS (
SELECT id, segment FROM network WHERE id = 6
UNION ALL
SELECT n.id, n.segment
FROM network n, walk_network w
WHERE ST_DWithin(ST_EndPoint(w.segment),ST_StartPoint(n.segment),0.01)
)
SELECT id
FROM walk_network




Which returns:

 id
----
6
3
1
(3 rows)


This can be made into a function like this:



CREATE OR REPLACE FUNCTION downstream(integer)
RETURNS geometry
LANGUAGE sql
AS '
WITH RECURSIVE walk_network(id, segment) AS (
SELECT id, segment FROM network WHERE id = $1
UNION ALL
SELECT n.id, n.segment
FROM network n, walk_network w
WHERE ST_DWithin(ST_EndPoint(w.segment),ST_StartPoint(n.segment),0.01)
)
SELECT ST_MakeLine(ST_EndPoint(segment))
FROM walk_network
' IMMUTABLE;




And here’s the function in action, generating the downstream path from segment 12.


=# SELECT ST_AsText(Downstream(12));

st_astext
---------------------------------
LINESTRING(4 2,3 2,2 1,1 1,0 0)
(1 row)

Monday, April 22, 2013

Crowd Sourced Information– Some Sources

Here are some useful links for accessing Crowd Sourced data:

Twitter

https://github.com/Twitterizer/Twitterizer

FlickR

http://flickrnet.codeplex.com/

Bike Data for Shared Bike Schemes:

http://www.citybik.es/    - check the API pages

  > Example of its use:  http://bikes.oobrien.com/london/

Tuesday, March 12, 2013

Useful things to know when using SQLite and .NET

This is worth knowing when using SQLite and the System.Data.Sqlite .Net library.

1) Various connection strings

Source:  http://www.connectionstrings.com/sqlite

e.g.

using cache>  Data Source=c:\mydb.db;Version=3;Cache Size=3000;

using a memory database> Data Source=:memory:;Version=3;New=True;

read only> Data Source=c:\mydb.db;Version=3;Read Only=True;

Journal files>
Data Source=c:\mydb.db;Version=3;Journal Mode=Off;
Data Source=c:\mydb.db;Version=3;Synchronous=Full;
(means full flush after each write)

Using compression >
Data Source=c:\mydb.db;Version=3;Compress=True;

 

2) DB locking issues and how to get around it

Source:  http://www.lastaddress.net/2010/11/systemdatasqlite-locks-database-even-on.html

using (SqliteConnection sqlCon = new SqliteConnection(connStr))
{

  sqlCon.Open();
  using(SqliteCommand sqlCom = new SqliteCommand(sqlCon, sqlStr))

  {

     using(SqliteDataReader rdr = sqlCom.ExecuteReader())

     {

        //do data stuff

        rdr.Close();

     }

     sqlCon.Close(); //CLOSE sqlCon BEFORE sqlCom IS DESTROYED!

  }
}

Sunday, January 20, 2013

Central Logger and Dashboard tool –Web App

I’ve been working the last 2 weeks on a central logging tool, and a web dashboard. The idea is that any component in a system can post a log record to the central logging server.

Log sentences are defined to be 4 parts long… these are message type (msgtype), message part 1 (msg1), message part 2 (msg2), and message part 3 (msg3). Each part is sent as a string. The server runs a concurrent memory queue, each TCP client sends a log… and the server time and task id are added at the time they arrive.

Every so often (eg 5 seconds or 100 new entries) the memory queue is cleared in a transactional update to the central SQL Lite DB. From testing it seems that this easily supports 10 concurrent users posting 10,000 records at the rate of 2000 inserts per second across the internet. Which is fine for what we need.

The second part is the dashboard, which I built using ASPX and a web service. It also uses OpenLayers to connect to a GeoServer instance to display map content. I use Flexigrid  (http://flexigrid.info/) to display the attribute data and allow for sorting and filtering.

I had 3 main issues with taking the running application from the dev environment to operational. These were:

1) the SQL Lite library supports 32bit and 64bit. I’d dev on a 32bit machine but was trying to run it on a 64bit server. I got around this by creating an IIS application pool and setting it to allow 32bit apps, then assigning this to the web app I’m deploying.

2) The service wasn’t available from any remote web browser, it only ran on the local machine (the web server itself). To fix this I had to add some extra lines to the webconfig…

inside of the  <configuration>   <system.web>
add this section

<webServices>
     <protocols>
       <add name="HttpPost" />
       <add name="HttpGet" />
     </protocols>
</webServices>

For more info:  http://support.microsoft.com/kb/819267

3) Everything was running fine on the IIS server…until after a while  the locationservice.svc was reported to be not be available from the browser.. the error the server reported was:

“Could not load file or assembly App_Web...”

So I copied the files across again to the IIS server and it would all be fine for a while again… until an hour or so later packing up again. This turned out to be an issue where the server recompiles the ASPX code every so often. To turn this facility off you need to add the following to the web config… under <system.web>

<compilation debug="false"  batch="false" />

For more on this error and the fix check this link:

http://stackoverflow.com/questions/4051576/could-not-load-app-web-dll-exception-after-asp-net-compile-recycle

Thursday, September 13, 2012

C# Parallel Convert a List<objects> to String with Custom Formatting

 

I wrote some code a while ago which did some processing, creating a list output, which I then had to turn into a String to TCP send it to a remote client.

It worked but took a while...

I was wondering why.... turns out the processing is quick as in parallel.. but the bottle neck was in the sending part...

Step 1... add a method to zip up the string and send that = slight performance increase as now sending a lot less data

Step 2... parallelise the list to string part...

It was like this.. and with about 40,000 items (egogridviewresult objects) in the list.. it'd take 20 or so seconds

for each (egogridviewresults v in r)
{
res += v.tarpt.X.ToString("0.00")+","+v.tarpt.Y.ToString("0.00")+","+v.cellx + "," + v.celly + "," + v.objid + "," + v.distance + "#";
}
res = Zip(res);
sendToClient(res);

- - - -  

I've now changed to this and it does the same job in about 2 seconds!

var results = new ConcurrentQueue<string>();

Parallel.ForEach(r, v =>
{
String r1 = v.tarpt.X.ToString("0.00") + "," + v.tarpt.Y.ToString("0.00") + "," + v.cellx + "," + v.celly + "," + v.objid + "," + v.distance.ToString("0.00");
results.Enqueue(r1);
});

res = String.Join("#", results);
res = Zip(res);
sendToClient(res);

__________________

Things learnt...

1) C# .NET 4 has a cool function String.Join which can concatenate a list into a string with a defined delimiter of your choice! 

2) Parallel.ForEach is just awesome!

Thursday, September 8, 2011

Raster access speeds… and size limits

 

I need to be able to handle large raster images, and to be able to find values from cells very quickly.

There are many ways to do this…. here’s a look at a few:

Method Handles GeoCoords Max Size Read Access Speed
DOTSPATIAL
Raster
Yes Medium (9k x 9k) 2.6 million per second
C# dictionary No Small (5k x 5k) 50 million per second
EMGU IMGs No Large (30k x 30k) 3.2 million per second
C# BMP (.net) No Medium (9k x 9k) 1 million per second
C# FastBMP No Medium (9k x 9k) 33 million per second
Rasterlite Yes Large (internal tiling) (to do)
GDAL Yes Large (to do)
R (grids) (to do) (to do) (to do)
XNA texture No Small   (4k x 4k) (to do)

Tuesday, September 6, 2011

DotSpatial

So I’m checking out DotSpatial for use in a project.. and the first thing on my list is to check it can convert coordinate systems easily from WGS84 <> OSGB36.

Seems the current version’s inbuilt definition of OSGB36 isn’t correct.. and when using it with the Reproject method gives out some strange results. It’s also listed as failing the unit test currently.

However you can specify the details yourself.. and then it seems to work just fine… so here’s the code for taking a coordinate as OSGB36 and transforming it to WGS84…

PointD pd = new PointD();
pd.X = 326097.07;
pd.Y = 673630.932;

double [] pts_xy = new double[2];
pts_xy[0] = pd.X;
pts_xy[1] = pd.Y;

double [] pts_z = new double[1];
pts_z[0] = 0;

var pStart = new ProjectionInfo("+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +towgs84=446.448,-125.157,542.060,0.1502,0.2470,0.8421,-20.4894 +units=m +no_defs");
var pEnd = KnownCoordinateSystems.Geographic.World.WGS1984;
 
Reproject.ReprojectPoints(pts_xy, pts_z, pStart, pEnd, 0, 1);
System.Console.WriteLine(pts_xy[0] + " " + pts_xy[1]);