Not all data comes quietly into your database.
📈 Finance and Accounting
There are certified, experienced professionals manually entering financial transactions into Excel off pieces of paper.This is not okay. It's a waste of expertise and easy to screw up. I can help you streamline that.
🧪 Alternative Data
Is there data on a website, spreadsheet, PDF, or in an streaming video that you regularly need updated in your database? I can get that set up as an automatic pipeline.
These are some examples of relevant projects I was responsible for.
USDA Market Data Report Parser
- Private project developed for a private firm
- Collects data from USDA reports that are human-friendly, human-typed text reports that are challenging to parse (above data is from here)
- Automated execution ensures user's database is always up to date
- Immediate e-mail notifications summarize new report contents with charts and statistics
Charity Job Hub
Multiple Sites, Unified Filtering
- Collected job postings from the largest charity and non-profit job posting sites and harmonized the job postings into a single database of listings
- Provided a filtering interface to users to select postings based on the relevance to their careers
- Supported both positive ("must have one of ...") and negative ("must not have any of ...") filters on all important columns
- Helped users keep track of relevant postings with a calendar of application deadlines and in-interface marking of "applied" and "interesting" jobs
- Updated daily and deleted expired postings (which many sites leave up for months)
Do you think you have a profitable trading idea, but want it independently verified? I have spent thousands of hours avoiding being fooled by randomness. Go beyond MetaTrader, MultiCharts, and Trading View's built in backtesting and get an independent report on the performance of your strategy. You'll get a devil's advocate to reign you in before you lever up. I'll give you an understandable analysis of your strategy from the perspective of a practitioner, not a statisticians report.
I created the data acquisition, analysis, and visualization system for a family office trading livestock futures.
It is a multi-threaded scraping system that uses a regular expression meta-language that I created to aid the parsing of USDA reports. The system also collects data using JSON, XML, XLS, and HTML scraping. FTP connection to the broker enables cross referencing futures data with account and position details. Automated emails notify traders of liquid spreads and contracts near expiry. A self-hosted (AWS) Tableau Server that I installed and configured integrates all of the above data. I know a little something about roll yield and basis and enough about options to handle your data.
I've spent countless hours making custom indicators and automated trading strategies, as well as building trading simulators and macro analytics systems. I haven't found a money machine yet, but I know how to avoid a false sense of security in backtested data.
If you have similar needs, reach out!
Python is the third most loved programming language according to the Stack Overflow 2020 survey. It's become a lot of new developers first language, and for good reason. It is easy to learn and has high quality libraries for just about anything. I switched from using Perl as my primary develop language to Python in 2011 because I wanted to leverage Python's modern code patterns and libraries and it stuck - for nearly 10 years. I developed many interesting tools, toys, and applications quickly and painlessly and had a great time doing it.
Today, I would say that Python's primary advantage is development speed. You can get your software feature complete quickly and it'll work. The interpreter stays out of your way and lets you approach your solution any which way you please. You want to write a function that can return a string, or an integer, depending on how you call it? Sure! You want to multiply a string by an integer? Sure, it'll do what you probably expect. Its dynamically typed interpreter and rich libraries make your job easy - at the start.
I switched from Python as my primary language because I built a large code base (~30,000 lines). Python got out of my way while I was writing it, but in hindsight I wish it slowed me down and forced me to write clearer code with better interface guarantees. I started adding type hints, but these are just aesthetic sugar you apply and are not enforced and did not address the challenge of refactoring. Static analysis tools exist that can enforce compliance, but I (possibly irrationally) do not trust their ability to fix what isn't ultimately broken; the language is designed to not get in your way.
I still use Python for data science in Jupyter notebooks and its much easier using pandas and matplotlib than it is to use Rust's ndarray and countless plotting libraries. My written, more serious projects are all Rust now, though. I will admit, systems programming and the nitty-gritty detail of memory has always been attractive for me and C was my first language. Python is probably great in other niches.
In 2020 (during the COVID-19 lockdown) I stopped fantasizing about eventually learning the Rust programming language and I started building projects. I've always been a systems programmer at heart, and as a enthusiast for the details of security vulneribilities and reverse engineering I always had safety on my mind when programming in C, Perl, Python, and elsewhere. The core focus on Rust is safety, and that really appealed to me. You get the low level power of C with modern code patterns. You get fearless concurrency as easy as .into_par_iter() and the confidence that if your code compiles, you didn't screw up.
I have found Rust difficult to learn and program in. The first stage is easy, you battle it out with the borrow checker and the helpful compiler gives extremely direct and useful suggestions on how to make your code work. There is an entire world waiting beyond that stage though: generics, traits, and some poorly documented crates (libraries). As a young language there are many libraries available for doing typical things, and its not often clear which you should use. Many of them are abandoned, many are poorly documented, and few are consensus picks. Sometimes you will find no analogous library that you relied on in Python and have to make it yourself.
With that said, I think this is my core language for the foreseeable future. The focus on safety forces me to write good code the first time instead of refactoring two years from now. The early development stage for my new projects is slower, but I suspect the total labor cost of development is actually lower due to the higher standards my code has to be held to initially. Libraries are improving, and I am making an active effort to contribute to that. The code patterns available to me in Rust are delightful. Match statements and patterns are beautiful. The functional programming influence encourages very elegant code. I can return to code that I've written and have a good idea of what it's doing and that there are no hidden side effects in my functions because it's mandatory.
Rust forces humans to write good code. I humbly accept these handcuffs, and am having a great time developing higher quality applications as a result.
- Code Example: Extracting Historical Data from Interactive Brokers API
- Compliment to Standard Move Risk Management: Volatility Bands
- Method to Sync MultiCharts Database + Studies Across Computers
- Function + Indicator for Risk Management: Standard Move
- Interactive Brokers Order Types and Algo Overview (For Ordinary Folks)