Stock Price Display Portfolio Using pandas-datareader

5/17/2021•pandas Portfolio React Django Beginner Engineer

1. Introduction

To test the results of studying on Udemy for a year, I created a portfolio before starting full-fledged personal development.
This is a web app where you can try the GUI for pandas-datareader, a Python library that can download stock prices and economic data. I incorporated popular technologies like React/mui/Typescript/Django/nginx/circleci/Docker/terraform/aws.

Self-Introduction

I used to do part-time work with Visual C++ but had been away from programming for a while, so web development is new to me.
After reading “Let’s Start Personal Development!”, I started studying React, Django, AWS, etc. on Udemy a year ago.

About pandas-datareader that I based it on

pandas-datareader is a Python library that allows you to easily download stock prices, GDP, exchange rates, etc. from various data providers.


In [1]: import pandas_datareader.data as web
        web.DataReader('^DJI', 'stooq')

Out[1]: 
                Open      High       Low     Close     Volume
Date                                                         
2020-07-06  25996.08  26297.53  25996.08  26287.03  344151697
2020-07-02  25936.45  26204.41  25778.12  25827.36  350387758
2020-07-01  25879.38  26019.31  25713.61  25734.97  374839860

2. App Overview

I created a web app so that even beginners who haven’t set up the environment yet can understand what can be done with Python and pandas-datareader through GUI.
It’s simple: log in, select a data provider in the sidebar, choose ticker (company code), start date, end date, press Submit, and the graph and table update.


Demo	Removed
Github	https://github.com/tanosugi/public_panta_watch

Time taken for development: About 1.5 months

Technology Selection Criteria and Impressions, Difficulties

Frontend

Selection criteria: I wanted to touch what is used in the world, so I chose React/Typescript.
Impressed by: Material UI and react-router are very easy to use once you get used to them.
Difficulties: Responsive design with Material UI (to display on mobile) works with sample code as is, but modifying it was difficult at first. Also, making it jump to the login page if not logged in with react-router was a struggle at first.

Name	Description
React/Typescript	Development without JSX is unthinkable.
Material UI	UI framework, used default designs for top and side bars, inputs, buttons, etc., used alpha version v5.
react-router	Routing library (screen switching), used beta v6.
Graphql	Used for communication with backend

Backend

Selection criteria: Since I wanted to handle stock prices and economic data, I chose Python/Django instead of Ruby on Rails or Laravel.
Impressed by: The wide coverage of each data provider, and pandas-datareader is wonderful because it can use them with similar interfaces. Black formatter is indispensable when writing Python. I tried installing PostgreSQL on EC2, but Docker is easier.
Difficulties: It took time to find a way to convert price data in Pandas dataframe format to JSON before sending via GraphQL. Docker builds take time, so I tried to use CircleCI cache but it was expensive, so I used Buildkit. I think the correct answer was to compress Docker cache and save it to CircleCI cache.

Name	Description
pandas-datareader	Download stock prices, GDP, exchange rates, etc. from data providers and provide to frontend
Django	Create Graphql API, implement login function, etc.
poetry	Virtual environment, installation seems faster than other virtual environments
black/flake8/isort	linter, formatter
nginx	web server
PostgreSQL	Django database
Docker/docker-compose	nginx, PostgreSQL are easier to use Docker than install, and since I was aiming for ECS Fargate, I containerized Django too. In the end, I decided to run these containers with docker-compose inside EC2, but this might be good for cost performance
Docker Buildkit	Since CircleCI cache is expensive, put cache in image

Infrastructure

Selection criteria: There are Heroku and Netlify, but AWS is the only choice for scaling, learning cost is high but no problem once you get used to it.
Impressed by: Terraform is the best because once it works, you can reproduce the same environment in development, staging, main.
Difficulties: Terraform was hard to get working.

Name	Description
AWS S3/Cloudfront	Static hosting for React/TypeScript
AWS EC2	Place containers for Django, etc.
AWS ECR	Place to store built containers
Terraform	Set AWS resources with Infrastructure as Code

CI/CD/Error Reporting

Selection criteria: There are Gitlab and CodePipeline, but CircleCI is standard in Japan, so I chose it.
Impressed by: With paid plan, parallel builds are possible, much faster.
Difficulties: Until I built the build method, every error required starting from the beginning, so it took time.

Name	Description
CircleCI	Set to build when committing to Github
Sentry	Error reporting from Django and React. Didn’t have time for integration with code map.

Development Environment

Name	Description
VSCode	Editor
EC2	Connect with Remote SSH and use

3. Impressions from Development

I learned that if you google, you can manage to some extent, and you can make something that works for now.
Unstable behavior is hard to solve without writing tests during development.
I re-realized that output is important because the technologies I modified from sample code while studying and used in this development are quite ingrained.
CI/CD seems convenient as it scales, meaningful even for personal development.
AWS was hard until I got used to it, but Terraform was even harder.

Reflections

I didn’t write tests, so errors aren’t completely crushed. (Excuse…)
Too many re-renders. (Should use React memo, etc.)
I wanted to challenge ECS Fargate, https with custom domain, but gave up because Terraform was hard (next time for sure…)
Design sense is important. Can’t apply even if you have sense without writing CSS, so CSS is also important, need to study (I thought I studied quite a bit…)

But please forgive me this time and use it for next time.

4. One Year Learning Record (= Udemy Course Record)

I learned general technologies on Udemy, by copying sample code and watching videos. I watched only about 30% of them, but I forgot after a month, so combination with copying was effective. I also referred to Qiita for individual items.
I counted and had taken 42 courses. My life was watching Udemy or copying it except when working and raising children.
I feel half (only ①-④) would have been enough, so I took a big detour. It might have been different if I had a mentor.
I started noticing the need for AWS and CI/CD after understanding React/Django, and finished understanding React/Django with Kazu.T’s course.

Learn TypeScript at the fastest speed [SNS edition] Full-stack web development with React Hooks + Django RestFramework API [Youtube edition] Full-stack web with React Hooks + Django RestFramework API [Test edition] Modern React software testing with Nextjs + React-testing-library [Basics edition] Full-stack web development with React Hooks + Django REST Framework API Complete strategy for web development software testing (React Hooks/Redux + Django REST API) GraphQL SNS function development (React + Graphene-django) React software testing (Modern testing methods in the era of Hooks+ReduxToolKit) [Instagram clone edition] React Hooks + Django Restframework Web app development with Firebase + React Hooks(TypeScript) [Redux edition] Modern React frontend development with Redux Tool Kit and React Hooks [COVID 19 app edition] Coronavirus Live Dashboard made with React [JIRA edition] Original JIRA made with React Hooks/TypeScript + Django REST API

Stock Price Display Portfolio Using pandas-datareader

1. Introduction

Self-Introduction

About pandas-datareader that I based it on

2. App Overview

Time taken for development: About 1.5 months

Technology Selection Criteria and Impressions, Difficulties

Frontend

Backend

Infrastructure

CI/CD/Error Reporting

Development Environment

3. Impressions from Development

Reflections

4. One Year Learning Record (= Udemy Course Record)

①Kazu.T’s React/Django Course (Absolutely recommend if you want to learn both frontend and backend ⇒ Instructor’s coupon)

②Python / Finance Courses (Combination of Python and Finance)

③AWS Solutions Architect (Studied and passed with below)

④React / Django / CircleCi / Terraform (Practical DevOps learning)

⑤Other React (Graph display, etc., acquiring basics)

⑥Other React/Django (Took all courses for React / Django combination)

⑦Design

⑧For a change of pace