Xtreme FME League

From Fmepedia

Xtreme FME League

Not long ago a user wrote to us that "the surface modeler is amazing, lately I processed 700 million points"!

I know there are some of you who would definitely consider FME-ing an extreme sport - so here is your chance to show off. Post here extracts from a log file that shows your xtreme use of FME in the categories longest translation (time), most features or most transformers.

Another category might be largest workspace file. I've spoken to someone with a 4mb fmw file. Anyone beat that?

We'll keep a top-ten of each category - and am sure we'll find some sort of prize for the winners.

Most Features Read

#
1
2
3
4
5
6
7
8
9
10

User
Dotted Eyes (http://www.dottedeyes.com/)
Hans van der Maarel
Fredrik Ronn, Stockholm Water CO.
.
.
.
.
.
.
.

Features
429,772,302
18,978,401
11,636,662
.
.
.
.
.
.
.

Comments
OS MasterMap to Oracle database using SuperpOSe (FME Objects) application
Overlay of 18.9 million points + 31000 areas - 2hrs 17mins
Read laser-scanned 3d-points from 247 csv files
.
.
.
.
.
.
.

Most Features Written

#
1
2
3
4
5
6
7
8
9
10

User
Dotted Eyes (http://www.dottedeyes.com/)
Fredrik Ronn, Stockholm Water CO.
Mark Stoakes
.
.
.
.
.
.
.

Features
429,772,302
11,636,662
353,499
.
.
.
.
.
.
.

Comments
OS MasterMap to Oracle database using SuperpOSe (FME Objects) application.
Wrote laser scanned features as a fanout to 18 datasets in SDE. 7 Hours.
Write to Oracle. Total time <2 hours = 52 rows/sec - damn that's quick!
.
.
.
.
.
.
.

Most Transformers

#
1
2
3
4
5
6
7
8
9
10

User
Mita Hajirakar
Mark Ireland
.
.
.
.
.
.
.
.

Transformers
802
167
.
.
.
.
.
.
.
.

Comments
AIXM reading workspace in Pro Services project
Demo workspace in training course has 167 transformers!
.
.
.
.
.
.
.
.

Largest Workspace

#
1
2
3
4
5
6
7
8
9
10

User
Mita Hajirakar
Mark Ireland
.
.
.
.
.
.
.
.

Size
9,901 kb
4,857 kb
.
.
.
.
.
.
.
.

Comments
AIXM reading workspace in Pro Services project
Demo workspace in training course is nearly 5mb in size!
.
.
.
.
.
.
.
.

Longest Translation Time

#
1
2
3
4
5
6
7
8
9
10

User
Dotted Eyes (http://www.dottedeyes.com/)
Fredrik Ronn, Stockholm Water CO.
.
.
.
.
.
.
.
.

Time
1166400 seconds (324 hours, 13.5 days)
25138.3 seconds (6.98 hours)
.
.
.
.
.
.
.
.

Comments
OS MasterMap to MapInfo TAB using SuperpOSe (FME Objects) application
FME time=7879.6 seconds - Think the rest was waiting for SDE database
.
.
.
.
.
.
.
.

Other Notable Translations

Producing maps with FME

I was recently asked to produce a number of small, simple maps (contour of The Netherlands with a certain rectangle indicated on it, each of the 57 maps having a different rectangle). Output was to be in Illustrator. So I set up an FME workbench to take a csv-file with the coordinates of all the rectangles and a simple country outline and write one EPS file for every rectangle. Total processing time: 11.4 seconds.

--Hans van der Maarel 04:48, 21 Sep 2005 (PDT)

Data Loading to Oracle Spatial

OK, hows this for loading times? 4 files of a total size of around 1.5gb coming from ESRI Shape into Oracle Spatial, nothing fancy in the load process just mapping from column to column.

If you want to read about the system this data is going into you can click here (http://www.scisys.co.uk/casestudies/government/cs_nfcdd.htm).

The FME log reported...

|Total Features Written                                                  353499
|=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
|Translation was SUCCESSFUL (353499 feature(s)/58694282 coordinate(s) output)
|FME Session Duration: 6785.8 seconds. 

This works out at 52 rows a second with a total time of just under 2 hours. Damn that's quick!

--Mark Stoakes 10:04, 3 Oct 2005

Dotted Eyes OS MasterMap Translations

Dotted Eyes have two products that make use of FME Objects functionality; InterpOSe (http://www.dottedeyes.com/software/os/translation/interpose.htm) and SuperpOSe (http://www.dottedeyes.com/software/os/translation/superpose.htm).

After some mutual efforts in solving a couple of memory leaks, and using a recent beta build of FME2007, these products were used to translate the entire national coverage of the Ordnance Survey MasterMap Topography layer. That's 429 million features, in case you were wondering!

Translation to MapInfo using InterpOSe:

• Input: 11,106 MasterMap (gz) files.
• Output: 284Gb of MapInfo Tab files.
• Number of features: 429,772,302
• Processing Time: 324 hours (13.5 days)

Translation to Oracle using SuperpOSe:

• Input: 11,106 MasterMap (gz) files.
• Output: Oracle database running on a Linux server.
• Number of features: 429,772,302
• Processing Time: 139 hours (5 days, 19 hours)
• Speed of Translation: 855 features per second

The machine doing the processing was a Dell Optiplex 210L, P4 3GHz, 1GB RAM, Win XP SP2.