Principal Software Development Engineer
idiCORE.com
idiCORE is a data fusion product which merges datasets from various sources into a massive database, capable of
finding precise data very quickly at large scale. I lead the design and implementation of the services and front end
layer, focusing on security, performance, and scalability. I headed a team of 4 to 10 developers, interfacing with
other technology teams, business owners and stakeholders, executive leadership, and external technology partners.
I frequently participated in strategy meetings discussing company goals and product opportunities. I was responsible
for interviewing and hiring developers on my own team and participated in interviewing and hiring decisions for
positions outside of my team, including technical and executive candidates.
Technologies used:
JavaScript, PostgreSQL, C++, AWS, node.js, express.js, vue.js, less
Highlights of my design and implementation:
-
Services were designed as a collection of independent worker modules that adhere to a common interface. This enables:
- Scaling at the worker level. Precise control over scaling out only the workers that require it.
- Flexible design. Modules could be implemented in different languages or on different platforms if required. This makes A-B testing different implementations very simple.
- One way data flow for security.
- Abstract communication and data storage from worker modules. This makes it easy to swap communication protocols without impacting module code.
- Redundancy for stability.
-
I implemented the services in node.js
- Promises to handle asynchronicity.
- UDP transport protocol between services.
- Shared functionality was kept in separate packages, facilitating reuse and cross-team collaboration.
- Node.js add-on, written in C++, to facilitate sharing a UDP transport and queueing load balancer module between systems.
- Data was made available on a customer facing API, utilized by our own web UI, various customers, and our internal batch systems, which accounted for several million searches daily.
- Data was filtered before being delivered to customers, by merging access rules from several sources, including federal and state privacy laws, customer industry rules, and contracts with our data providers. We ensured that no customer received any data it was not allowed to view.
- Authentication system providing authentication tokens from stateless servers, using JWT (JSON Web Tokens). Node.js web server to handle UI login, as well as service authentication, mobile authentication, and internal batch system authentication.
-
Authorization system.
- Authorization rule inheritance, allowing for easy organization, and also fine control of individual exceptions.
- Effective permissions were included in the authentication token, stored in a proprietary binary format which allowed us to keep the overall token size small, while still including full detail of all permissions.
- Customer database storing relational data for customers, users, passwords, authorization rules, and preferences.
- Administrative tools for customer management, including self-service customer facing tools.
- Web gateway server with outbound, but not inbound, internet access. This was used as a web proxy for our web servers, which were exposed on the internet, but lacked any outbound internet connectivity. This enabled us to isolate systems, narrowing the attack surface and protecting against data breaches.
