Making Selections Based on Color and Focus

Before

After

I was able to change the the color of the toothbrush as well as the background into a more saturated color by selecting the desired shared color within the entirety of the image. This in turn is able to change the tone of the photograph taken as I am able to darken or lighten colors that I want as well as being able to focus in on certain subjects within the image. I was able to make the toothbrush stand out more by changing the color of the tooth brush and the toothpaste, draining its color almost entirely, whilst the background adds  more emphasis on the subject of the photo. This technique can be used as a means to change large portions of background colors to either lighten or darken the photo, thus changing how colorful or monotone it is.

Using Select and Mask to Refine a Hard Edge Selection

Before

After

Using Select and Mask, I was able to isolate the squash and change its color into a brighter yellowish shade of yellow.  This process makes it easier to change colors using the quick select tool as well as the masking tool to select as much of an image as accurately as possible. This technique can be used in the future if there is a smaller, more particular object you want to edit it's color with within a crowded image, such as a forest, or a city block.    

Selecting Subjects

Before

After

;This week I learned about selecting objects and editing them into other photographs. I was able to use some of my previous knowledge on isolating an object in a photograph (see previous blog). I was then able to use smooth soft edges from his hair in order to prevent unwanted parts of the background. I then used another photograph I took and added it as a background image to complete the photo. This week's practice makes me want to learn about how to shade in objects in order for them to blend in with the photo more seamlessly.

Using Marquee and Lasso Tools

Using the lasso tool, I was able to create body parts for the character above. In order to create y character, I made sure to use multiple layers if I wanted to move any body parts or erase certain parts of it that didn't look right. I am able to  use these tools in the future if I want to edit the characters or photos using drawings of my characters, I can use the lasso and marquee tool to manipulate certain parts of the photo I want to edit.

Soft Edges

Before 

After

The photograph of the teddy bear after is a demonstration of how I utilized the select and mask feature to isolate the bear from the background, maintaining it's individual hairs on the object. This process was fairly easy once I understood what was needed to do as I was able to select and deselect  the object of interest. Masking the rest of the background was also fairly simple, brushing over the edges, softening it in order for it to make the object less rigid. I will definitely use this technique in my future works.

Quick Selection

    

Before using Quick Selections

After using Quick Selection

I was able to use quick selections to change the hue of the penny by selection the penny alone and adjusting the color to another one. When photoshopping the penny using quick selections, I noticed how easy it was to select larger areas that would take more time to fill in compared to smaller areas, which automatically filled in unnecessary spaces. The quick selection tool makes me wonder how I am able to use this feature for more complex images in the future

Documentation Through Photography

After documenting my daily life through photography, I discovered that my life is very routine-like, as looking back through them all, I noticed how I'd do the same sort of things every weekday. The greatest challenge of choosing only 12 out of 24 photos to add to my final essay was how I had good photos from one day, and another equally good photo from another day that was exclusive to that day, such as percussion club and drama club, or when there was snow one one day, but there was none on another. I decided to follow the guideline of consistency, in order for it to appear as if the photos were all taken in one day. Dorothea Lange and her photography influenced what I took pictures of as I simply seized the moment of opportunity when I saw a potentially good photograph. rather than simply doubting myself.

Engineering Journal

I have begun to create a website in order to help users navigate the PATH system. I am using the website known as Weebly and utilizing information found on both the official PATH website and Wikipedia. I plan to incorporate a number of features. I have finished the History section, which details the history of the Subway system. I am attempting to work on the other sections such as Directions and Station Information. The Station Information section is going to require some field work and testing. What I mean by this is that I will need to go out to all 13 PATH stations and gather information in order to put this information in this section. The Directions Section will probably be the hardest as I need to figure out how to program the buttons in order to respond to a combination of two buttons and then write a script for how to get from Station A to Station B for each combination. While there are 169 different combinations theoretically, that includes when you input the same station twice, and going between 33rd and 23rd as well as 23rd and 33rd for example. It should not be too hard, but I need to figure it out. I also have this Gantt Chart in order to show how I am going to work on the different features of the website. One can find it in the Google Drive link

11/5/18

Today the group made gradual progress on the project. Max and Joy learned Python and practiced on Youtube. Sophia worked on updating her software, which has proven to be a tedious task. We will continue to practice coding and updating software, hopefully all software will be up to date by Wednesday and the app can begin to develop. The only problems encountered was that the software previously installed on the computer is so out of date that to update it one has to do it in small increments. However with time, this will be an easy fix.

Input Features 

Input Features The input features consist of three sets of variables.

1. The first set is the historical daily trading data of INTC including previous 5 day’s adjusted closing price, log returns, and OHLC variables. These features provide basic information about INTC stock.
2. The second set is the technical indicators that demonstrate various characteristics of the stock’s behavior.
3. The third set is about indexes: S&P500, CBOE Volatility Index, and PHLX Semiconductor Sector Index.

Category 1. Daily Trading Data

• S&P 500
• An American stock market index based on the market capitalizations of 500 large companies having common stock listed on the NYSE/ NASDAQ
• VIX
• The CBOE Volatility Index: a popular measure of the stock market’s expectation of volatility
• SOX
• The PHLX Semiconductor Sector: an index composed of companies primarily involved in the design, distribution, manufacture, and sale of semiconductors

Other performance measures

1. Allocative efficiency
1. Total surplus (welfare) is my key measure of market performance. Welfare indicates how well the market allocates trades according to underlying private valuations.
2. Liquidity
1. Markets are liquid to the extent they maintain availability of opportunities to trade at prevailing prices.
2. In other words: Liquidity is defined as the ability to exchange an asset for money at a price as close as possible to the equilibrium price
3. Price discover- This reflects how well prices incorporate information.

Literature Survey

Quantitative investment (QI) products (models/tools/systems) can provide accurate stock market prediction and help investors significantly alleviate risks of mispricing and irrational trading because of psychological factors, such as overconfidence, mental accounting, loss aversion, and so on

Article: A causal feature selection algorithm for stock prediction modeling

Zhang, Xiangzhou, et al. “A Causal Feature Selection Algorithm for Stock Prediction Modeling.”

NeuroImage, Academic Press, 9 May 2014,

www.sciencedirect.com/science/article/pii/S0925231214005359.

The main issue of quantitative investment (QI) is what features to include. This article uses algorithms in data-based analysis to enhance the QI. It also states that CFS (casual feature selection) algorithms are the best predictors. They are able to identify the variable and generate a feature subset based on the results. Other algorithms in existence are principal component analysis (PCA), decision trees (DT:CART), and the least absolute shrinkage and selection operator (LASSO). CFS is the most accurate and precise and has the ability to develop the QI product. Other common stock algorithms can only reveal base level detail and not stock features (inputs) and stock return (output). CFS’s have two unique aspects, one being to identify direct influences between variables and to verify the algorithm using extensive experiments (accuracy, precision, sharpe ratio, sortino ratio, information ratio, and maximum drawdown (MDD). PCA can reduce a large set of variable to a smaller set of uncorrelated factors while keeping variance. The results of PCA is a subset of original components but not of variables, making the resulting data hard to comprehend. DT is one root (node) and a number of branches. With only features that contribute to the classification. DT uses the entropy theory that selects specific variables with the most important information. LASSO which selects individual variables.

Additionally, there are two types of stock prediction categories, time series forecasting and trend prediction. A time series forecasting model is trained to fit the historical return/ price series of individual stock, utilised to predict the future return/ price. A trend prediction model is used to obtain the relationship between various fundamental and technical variables and the movement of stock price. Dataset, prediction models and selection algorithms include SRA, PCA, and genetic algorithm (GA), informational gain, etc. Inputs vary but often both technical and fundamental variables exist such as economic variables.

Other data mining algorithms exist such as: logistic regression (LR), neural network (NN), support vector machine (SVM), decision tree (DT). Supplementary algorithms include Bayesian network (BN), Zuo nad Kita.

Feature selection can be grouped into two categories, filter and wrapper approaches. The filter uses general characteristics of the training data to select key input features. Wrapper uses the prediction performance of a special learning algorithm to evaluate and determine the best feature subset. Evolutionary algorithm and GA are used in the latter, allowing it to perform better, however it is more expensive.

Article: Using Bitcoin Pricing Data to Create a Profitable Algorithmic Trading Strategy

Link: http://cs229.stanford.edu/proj2017/final-reports/5244395.pdf

Uses data from historical GDAX prices which was able to output the current open, close, high, low price and volume every minute time interval over the past year approximately 450,000 data points. This projects goal was to buy and sell in x minutes through predicting the ratio of the price x minutes later to current prices rather than predicting a standard up/down value.

The features they used were High/Current price, Low price/Current price, Average price/Current price, volume of trading in BTC, Proportion of increase in price every minute, proportion of convex change every minute, ratio of price n minutes ago to current price, average price/current price, volume n minutes ago/current volume, etc.

(PCA and Feature Selection) WAP, VR, and R are not viable options. WAP is less varied than AP while being highly correlated. VR has low variance and an inaccurate predictor. R was similar to A however, A captured more information. Meaning, Baseline, LinReg, LonReg PCA and Neural Networks remained.

The methods that were used were Baselines, Weighted Logistic Regression, Principal Component Analysis and Neural Networks. Weighted average, gains and AUC (area under curve which measured the true positive rate vs the false positive weight) were used as measuring increments. *Refer back to article for indepth*

In conclusion, all of the listed, approved models had gains that outperformed the average increase per minute of bitcoin. Meaning that actually using them in market like situations was more effective than buying and holding. PCA had small increased gains but with conjunction to neural networks, significant gains were represented. Their algorithm was questioned to whether or not it was actually 100% reliable due to recent spikes. *This was not tested in real time*

Article: Price Prediction Evolution: from Economic Model to Machine Learning

Link: http://cs229.stanford.edu/proj2017/final-reports/5241834.pdf

Used the stock index to gain data and comprehensive market tendency. They extracted four features: Max, Min, Mean, and Standard Deviation every 5 days. They inputted the data by a Shifting Window Pattern. They first ran multilinear regressions adding features one by one. The input features they used were univar, bivar, CPI and GDP. Using GDP as the main predictor of the closing prices. They regretted using increments of 5 days as it may not have reflected the real behavior of the market. Concluding that the more macroeconomic features they used in their algorithm diluted their results. Recommending: Using only closing prices and certain macroeconomic features as a good approach.

LWR is used to predict certain features in the market. Defeisiancis of LWR is that it has time lag which then wont reflect the most up to time data. However, it is one of the most accurate predictions. NN used the max, min, mean and standard deviation to feed into the neural network. WIth 2 hidden layers activated by ReLU and no output activation layers. Using cross validation to tune the parameters. They then split the data into the train set, dev set and test set. Importantly using the dev set to adjust the models topology (number of layers, neurons per hidden layer, size of mini batch). Outputting 500 data with 481 MSE plotted into the data. SVR (support vector regression) overcomes difficulties in dimensions. Developed by SV and used it to predict stock behavior (forecasting the curve tendency). SVR is more flexible as it uses relaxation variables. In SVR they used libsvm in MATLAB and divided data into 3 groups 4000(train), 500(dev), 500(test).

Article: Application of Deep Learning to Algorithmic Trading

Link:http://cs229.stanford.edu/proj2017/final-reports/5241098.pdf

Project was based around LSTM (Long Short Term Memory) Networks. Their algorithm predicted stock data with the goal to forecast the next day’s stock price of Intel Corporation (NASDAQ: INTC) by adjusting the closing price based on information/ features available the next present day. Trading Intel stock accordingly to the strategy developed. Mostly using Locally Weighted Regression.

The Trading Framework used had 4 steps. (1) Input: used daily trading data, technical indicators and indexes. (2) Model: LSTM Network, LWR model. (3) Output: Predicting next day’s price. (4) Decision: Trading decision of whether to buy or sell.

Variables used: (3) historical daily trading data was composed of INTC’s last 5 days adjusted closing price, log returns, open/close price, high/low price, trading volume. The second set was composed of technical indicators that demonstrated various characteristics of the stock behavior. The final set included the S and P 500, CBOE Volatility index and PHLX Semiconductor Sector.

The technical indicators used were the Rolling Average/Standard Deviation with 5 and 10 days window, Bollinger Band: two standard deviations from a moving average, Average True Range: a measure to volatility of price, 1 month Momentum: the difference between current price and the price 1 month ago, Commodity Channel Index: an identification of cyclical trends,  Rate of Change: the momentum divided by the price 3 months ago, Moving Average Convergence Divergence: a display trend following characteristics and momentum characteristics, Williams Percent Range: a measure of the buying and selling pressure

*Reference for model, data, outcome, price prediction and return plots*

Strategy: used trained models to compute the predicted price. If the price the next day was higher than the current price, one share of INTC was bought. If the predicted price of INTC is lower than the current price, one share was sold.

Conclusion: LSTM Network and LWR models can predict the general trend of the INTC stock price. LSTM outperforms LWR in terms of profitability and accuracy over 3 periods. LSTM is more robust than LWR. LSTM has a smaller MSE than LWR for Dev set and Test set. LSTM yields higher returns and Sharpe ratio than LWR based strategy and simple buy and hold strategies. *Exclude dramatic price changes*. Using hyper parameters and adding regularized term would improve the performance and reinforced learning could generate more stable and higher returns.  

*Reference report for LSTM equations and explanation

Article: Predicting stock prices for large-cap technology companies

Link:http://cs229.stanford.edu/proj2017/final-reports/5237355.pdf

Used data from previous days and financial news articles to predict the changes in the future of a given stock. Prediction was excluded 10 days before and after the company’s Earnings report dates to reduce dramatic changes. Overall, this algorithm was 59% accurate and annualized a return of about 15%.

Used NASDAQ(.com) to gain data from 5 years ago. News data was pulled from XIGNITE(.com) and are processed to remove duplications and have exact dates assigned to each headline.

To digest the news articles, Naive Bayes theory was used which modeled 400 days and tested on 200 days of data. The output is the probability of a positive relative price change. Fixed tokens were used as replacements to numbers, percentages, and money amounts.

Features of the NN model: Relative price changes for the same stock from the past 20 trading days, News predictors from the last 20 trading days, news predictor of the current day. The results show that percentage gain is the most important factor. This project has 56.35% accurate predictions with news and without news resulted in 52.38%. NN w/ news produces the most accurate results with 59% accuracy and a daily gain of 0.0423%

In reflecting the author realized that stock prediction was too complex to be captured by price changes and news. XIGNITE(.com)’s news could have had higher quality news. Reducing and removing stemmed words into base forms, insignificant words, etc would improve prediction accuracy. And including trends in industries, political influences, competitor trends, etc to capture all the complexities of a stock market.

Article: Using AI to Make Predictions on Stock Market

Link:http://cs229.stanford.edu/proj2017/final-reports/5212256.pdf

Goal is to create an automated tool for managing investments with limited amounts of stocks. First, they designed an algorithm to predict increases and decreases in the next n days, using the stock prices and volumes in the past m days. Alpha Vantage API was used to access the daily open price, high price, low price, close price and daily volume since 2000.

Predictions based on Technical Indicators: Moving Average Convergence Divergence (MACD), Stochastic oscillator (STOCH), Relative strength index values (RSI), Average directional movement index (ADX) values, Absolute price oscillator values with SMA (Apo-SMA/ APO-EMA), Commodity channel index values (CCI), Aroon, Bollinger bands (BBANDS) values, Chaikin A/D line (AD) values, On balance volume (OBV) values.

Results: The variable targeted was the difference between the prices over a n-day period using the predictors: the stock price at the end of the day, beginning of period, volume of the same day, and the 11 indicators from the previous section. Using the training set, dev set, and test set in a ratio of 7:2:1. And evaluating the data with classification (of the price trend (increase/decrease)) and the mean squared error. Data shows that the support vector regression gives the best result however, it is expensive. So, for larger data being processed, linear regression is used.

Article: Optimised Prediction of Stock Prices with Newspaper Articles

Link:http://cs229.stanford.edu/proj2016spr/report/010.pdf

Explored the predictive power of using newspaper articles on the stock prices of companies. The testing accuracy was up to 61% using supervised learning. Additionally, the prediction algorithm ran through MDP (Markov decision process) which buys and sells shares on a simulation that is programmed. Binary responses along with machine learning and newspaper articles were used to predict up or down changes in stock prices. Past algorithms include Naive Bayes algorithm, SVM, the Perceptron, Boosting and bag of words. In addition, they expanded the bag of words to include shorter and more words. Term frequency-Inverse Document Frequency (TF-IDF) was used to improve the algorithms ability to grasp words and cross validate. Afterwords, a Markov Decision Process (MDP) learner was created that builds the base predictor and reinforces its ability to buy and sell shares.

Previous work includes (1) Gidofalvi: implemented naive bayes text classifier on financial news to track short term fluctuation prices, concluding that there is a strong correlation between news articles and stock prices in a 20 minute window before and after the publishing of the article.

Data: accessing New York Times and the Wall street Journal was difficult do to previous set up guidelines that hindered website access. Making the newspaper research reliant on Proquest Newsstand database. The input went from item term searches to federated search algorithms to PQNS XML TREE to XML tree parsing algorithm to PQNS article URL to Article Parsing algorithm to raw text to stemming algorithm to stemmed text to output.

Proquest newsstand archives all articles to a searchable database. With a written algorithm to generate federated search URL’s you can automatically obtain PQMS XML Tree which contains the URLs of the full texts of articles that mention the company of interest. Writing an XML tree parser will allow data to automatically be compiled into a list of PQNS URLs where the full texts are contained. Proquest needs a validating URL, meaning that there must also be a web scraper specifically to work around limitations of the Proquest database generating cURLs so that the article algorithmic gatherer can act like a real user’s browser. The use of regular expressions to scan full texts from webpages to obtain raw data to go to the Python stemming library for pre-processing.

Article: Using News Articles to Predict Stock Price Movements

Link:http://cseweb.ucsd.edu/~elkan/254spring01/gidofalvirep.pdf

Short term stock price movements can be predicted by financial news articles.