I live for solving difficult, real-world problems using
mathematical principles and solid computing programming
techniques. I believe most phenomenon can be accurately
modelled mathematically and realistically simulated in
silico. I am seriously passionate about innovation
and thrive in environments where cutting-edge
business-related research and development is being
undertaken.
I have 12 years experience in software research and
development for next-generation, extreme-scale
high-performance computing (HPC) systems and massive scale
data analytic and throughput methodologies. I apply
these skills learned in every task I have put in front of
me.
I am now getting very close to a version 1.0 release of my software. It has taken quite a lot of effort to get it to this stage and has required me to solve quite a lot of very difficult issues along the way (you should see the performance of the new ray-box intersection algorithm I came up with – realtime graphics now available on a cpu). I’m currently filing global patent applications for quite a few things (on the advice of a few interesting people).
Shearspace provides a simple way to perform massive scale, real-time visualisation of datasets all within a HTML browser (no extensions, no WebGL, no GPU, purely native browser). It can deliver FullHD (1080p) resolution at 30 frames per second for total interaction of monster sized datasets. The nature of Shearspace also enables people to share their datasets with stakeholders but not compromise their data security. That is, if you own sensitive data, you can allow specific people to view that data without ever having to give them a physical copy. This helps on two main fronts: data security and the difficulties in moving very large datasets around. You can set a variety of rules regarding the extent stakeholders may view data.
Shearspace loads data files natively as it does not require specialised data formats. You can thank years of HPC performance programming for that one. Data files can be stored in AWS S3 at the standard pricing model (typically 3 cents per GB per month). As a result, you are not limited to just visualising a single data type; You can view RGB, Intensity, point returns, classification, scan angles, height; The full range of data. These data types can be manipulated with a user defined 1 dimensional transfer function for interactive colourmaping.
The renderer itself is capable of drawing in excess of 12 billion points per second (single core). This performance will scale proportionally as modern cpu architectures progress as well as being multi-core (but I only use one core for the rendering at this point because that is all I need). Of course, given the remote visualisation nature of the software and that the rendering resources can be located thousands of kilometres away, rendering speed is the least of the problems. I think that the lag, bandwidth and other network problems have been solved nicely. Higher geometric primitives are also ready but not flagged for version 1.0 at this stage.
People will no longer have to purchase expensive graphics workstations and deal with expensive, and restrictive, software licenses that require yearly maintenance. Shearspace is available on demand with a very cheap pay-by-the-hour model. This enables clients to cost optimise their IT infrastructure.
I’m not sure yet on the absolute release date into the global AWS cloud as I’m still stress testing the performance of encryption and machine scaling (I have to ensure that my load balancer/monitor can handle up to at least 10,000 AWS instances, globally located, seamlessly).
If anybody would like to have a look and play with a pre-release version, just email me at the contact details at the bottom of the Shearspace website. I’d like to thank QCIF and QRIScloud for providing me with resources to enable client testing.
I’ve had a lot of fun writing this software. It is really nice to devote fifteen months of effort into an idea of remote visualisation that I’ve been mentally cooking up since 2003 (my first IEEE remote visualisation paper, on mobile devices before smart phones existed!) and utilises everything I’ve learnt during my education and working career (20 years in total).
Flagged for version 2.0 release is the point cloud classification algorithm. Shearspace has been built with a scalable, parallel, computational engine that enables highly computational algorithms to be applied to the point cloud. This is something that vendors cannot do with even powerful workstations. These algorithms require potentially 100’s of compute hours to deliver accurate results. Shearspace allows users to scale up their computational requirements simply, cheaply and on demand. I hope to also develop the ability to extend into Amazon spot pricing (100 compute core hours for less than a dollar). But my classification engine will take next priority.
MLS with Road Noise (traffic) | MLS with Automatic Noise Removal |
[Click for Slideshow] |
I'm responsible for leading the research and development associated with providing solutions for dealing with massive scale and complex spatial datasets. I apply strong mathematical and computer science skills and methodologies in the areas of data storage, feature extraction, analysis, visualisation and simulation that enables Roames to deliver scalable and optimised technological solutions.
Key Deliverables
500 million, double precision 3D
points of Brisbane CBD created by automatic
extraction of waveform laser data [click to
enlarge]
|
My job was to liaise, consult with and support QUT researchers across a range of areas relating to the application and support of high performance computing infrastructure to research problems. This included practical support to researchers and research projects including consultation and advice, specialist training, and tasks relating to the management and development of the IT research environment, including virtualised services. Support for both compute-intensive and data-intensive research codes requiring scientific code optimisation was a focus for this role (e.g., memory management, code optimisation and parallelisation)
Primary Responsibilities
Provided research support to QUT institutes, faculties and individual researchers by:
Supported researchers involved in the development of high performance research codes and applications (eg compute- and data-intensive applications) on infrastructure available by having:
Provide consultancy and support of high performance computing tools, applications and scientific libraries by:
Optimisation and parallelisation of numerically intensive codes, scaling to multi-rack, multi-network compute clusters. Parallelisation strategies range from simple vectorisation (full utilisation of SSE instructions) to full implementation of socket level programming for communication via numa or infiniband coupled with pThread implementations.
I have ported/tuned/parallelised more than 650 floating point intensive codes in the last seven years. Some projects, engineering simulation for example, I have had 800x speedup after 45 minutes of code inspection. Another example for Computational Statistics was to reduce runtime from 20 years to 24 hours (7,300x speedup).
Randomly
Selected Month of Savings Achieved by Code
Optimisation
|
The types of codes include, but not limited to, Econometrics, Non-linear Algebra, Computational Statistics, Information Systems, Particle Physics, Bio-Mechanical Engineering, BioInfomatics, Business, Geographic Systems, Accountancy, Electrical Engineering, Structural Engineering, Geology and Mining.
Measuring Cache Latency
|
SIMD Registers
|
Core/Cache
Locality
|
Highly capable of processing extremely large datasets (3 Petabyte compressed dataset largest so far) in theoretically minimal elapsed times. That includes development times. Design and architecture of tailored or "off-the-shelf" hardware using zfs, nfs, ext4, xfs filesystems.
Worldscope (including automatic update), LiDAR (GIS, extreme scale point clouds), 10 year Australian Electricity spot prices, Wikipedia corpus, World Intellectual Property Organization Patent Applications, Synchrotron, are but a very few examples.
While dealing with large data, it becomes necessary to examine data, dataflow and datastructure in meaningful ways to reveal hidden structures not apparent with algorithmic analysis. I've always enjoyed employing numerous techniques to generating scientific visualisations and am quite passionate about it. I am very well versed with GTK, GLX, OpenGL, SDL and VTK programming libraries for generating spectacular images, videos and interactive programs on a plethora of viewing environments. In my spare time, I even work on my own visualisation systems.
Interactive Native Megapixel
Displays of Massive Scale Data (Own software
libraries)
|
Interactive, Large Scale Volumes
|
Highly proficient down to bit manipulation level. Parallelisation using any technologies but mostly versed with MPI/OpenMP/pThreads/TBB/IPP. Preference for Intel compiler (on Intel hardware) and GNU compilers. Very familiar with profiling tools; VTune (amplxe-xxx)/gprof/codecov/valgrind/inspxe-xxx/etc.
Well versed on CPU architectures (past, present and future), including the number and types of registers available and how to program to use them without resorting to assembly code. This method enables programs to scale to new processor architectures without having to rewrite code.
Have delivered many training courses over the years on
scientific programming with C++ and 'bare metal'
programming. These training courses have inspired
and encouraged programmers new to the challenges of the
"Big Three": Big Compute, Big Data and Big Visualisation.
Have developed numerous algorithms in all areas of
Science, Technology, Engineering and Mathematics.
These algorithms have required writing from the ground up
since most available libraries are inefficient, resource
hogging and don't take advantage in recent cpu
advances. Total design of data structures to be as
light weight as possible yet extensible to take advantage
of various cache line sizes and cpu registers.
Strong exposure to Fortran due to 'old school' Professorial academics. Experience with tuning, parallelising and writing checkpointing code for those simulations that can run for years.
Intimate knowledge of Matlab and its toolboxes. Frequently ported codes to make use Mex files, the Matlab compiler or the parallel and distributed toolboxes (including GPU implementations). When performing exploratory algorithm selection/implementation I turn to Matlab simply due to increased speed of development.
I frequently blog about using Matlab, Mex files and the Intel Compiler and Intel Libraries and tools.
Being a Linux user, I primarily turn to bash (sed, awk, make, etc) and its abundance of command line tools to achieve quick turn around times for project delivery. Bash (or command line scripting) is often overlooked as a way to achieve goals in quick yet reproducible ways.
I've used Perl extensively over the years to perform all sorts of tasks. I have found it particularly useful using it to talk via a web interface to underlying cluster submission systems. Although, not computational, Perl provides many advantages in terms of quick analysis and processing of input data. Advantages of perl are that it is easy to get a client to use since portability is nearly universal.
Not an ideal language choice for computational programming but have used Java extensively for web delivery of services and simple system monitoring tools (It should be noted that these language advantages have been largely superseded by HTML5/JS/CSS programming. Servlets, Graphics, Mobile Devices, are some programs that have written and delivered for clients/projects.
This is largely a new paradigm but it is easy to see how powerful HTML5 will be in delivering services and results, particularly computational results. Some of the biggest problems with delivering services have been how to make a program easy to use by those with not so great computer literacy. HTML5 is a way to hide difficult OS particularities and allow easy communication with a back end service (on a supercomputer or some other powerful/appropriate hardware) to deliver the final result.
I have begun to port many existing services to use the full capabilities of HTML5, particularly the Worldscope database, other financial databases and GIS database frontends. It is also quick and easy to provide simple visualisation services (particularly information visualisation) to these existing services for quick perusal of data.
There are many other languages that I know but cannot
even recall using in the last 6 months. I have listed the
main languages above that achieve most things that are
required to be solved using computer languages. I've
regularly worked with R, Python, Ruby, LISP, C#, Basic, VB
(and their associated libraries and frameworks) to name
but a few. I have even solved problems for people in
languages that I've never heard of nor can I recall the
name. Programming is about a state of mind and a way of
thinking; Not about the language. This is why it is
referred to as 'The Art of Computer Programming'
Studied advanced calculus, linear algebra, vectors and matrices, computational mathematics, differential equations. Specialised in applied mathematics, scientific computation and visualisation.
Majored in software engineering. Studied data warehousing, digital environments, enterprise systems, network systems and web technologies.