Trend Analysis Review: Structure & Insights

1. Spotify Integration

In this project, I selected playlists from well-known Spotify creators, focusing on their relevance to trending music and social media engagement. This process led to a collection of 866 tracks. These playlists were chosen not only for their popularity but also for their relevance in the context of content creation, making sure they reflect the kind of music that resonates with listeners across various platforms. To keep the data accurate, I regularly update the popularity metrics to capture real-time changes in trends.

The track data was pulled in JSON format from Spotify's API, which provides detailed information such as popularity, energy, and tempo. These attributes are key for understanding how music trends evolve and what kinds of songs are gaining traction among listeners.

Spotify Playlists

Sample Spotify JSON

            {
                "album": {
                  "name": "Album Name",
                  "external_urls": {
                    "spotify": "https://open.spotify.com/album/2up3OPMp9Tb4dAKM2erWXQ"
                  },
                  "images": [
                    {
                      "url": "https://i.scdn.co/image/ab67616d00001e02ff9ca10b55ce82ae553c8228",
                      "height": 300,
                      "width": 300
                    }
                  ]
                },
                "artists": [
                  {
                    "name": "Artist Name",
                    "external_urls": {
                      "spotify": "https://open.spotify.com/artist/artist_id"
                    }
                  }
                ],
                "name": "Track Name",
                "popularity": 85,
                "external_urls": {
                  "spotify": "https://open.spotify.com/track/track_id"
                }
                ...
              }              
        

            {
                "danceability": 0.8,
                "energy": 0.7,
                "key": 5,
                "loudness": -5.0,
                "mode": 1,
                "speechiness": 0.05,
                "acousticness": 0.1,
                "instrumentalness": 0.0,
                "liveness": 0.1,
                "valence": 0.6,
                "tempo": 120.0,
                "type": "audio_features",
                "id": "track_id",
                "uri": "spotify:track:track_id",
                "track_href": "https://api.spotify.com/v1/tracks/track_id",
                "analysis_url": "https://api.spotify.com/v1/audio-analysis/track_id",
                "duration_ms": 180000,
                "time_signature": 4
              }              
        

I use scripts like spotify_api.py and spotify_insertion.py to extract the JSON data shown above and import it into my Django models, which are stored in a MySQL database. This setup allows for easy data management and supports in-depth trend analysis based on the musical attributes collected. Additionally, I’ve set up scheduled tasks to run these scripts and others automatically, ensuring that the data is always up to date without manual intervention.

Sample Track Model

Sample TrackFeatures Model

Sample Popularity History Model

2. Features & Calculations

This project leverages two types of features: Spotify attributes and calculated variables. The Spotify attributes are retrieved directly from the Spotify API, providing data points like tempo, energy, and popularity. After initial analysis of these attributes, valence, tempo, speechiness, danceability, and liveness were identified as the most relevant features for trend analysis. By combining these key attributes with calculated variables such as velocity and trend, we can effectively assess the importance of each feature in the model selection process.

Calculations

                            historical_popularity = self.get_historical_popularity()
                            if len(historical_popularity) < 2:
                                self.velocity = 0
                                return
                            
                            recent = np.array(historical_popularity[-2:])
                            rate_of_change = (recent[1] - recent[0]) / recent[0] if recent[0] != 0 else 0
                            self.velocity = rate_of_change                          
                        

                            historical_popularity = self.get_historical_popularity()
                            self.median_popularity = np.median(historical_popularity) if historical_popularity else None
                        

                            historical_popularity = self.get_historical_popularity()
                            self.mean_popularity = np.mean(historical_popularity) if historical_popularity else None
                        

                            historical_popularity = self.get_historical_popularity()
                            self.std_popularity = np.std(historical_popularity) if historical_popularity else None
                        

                            if self.velocity > 0.1:
                                self.retrieval_frequency = 'high'
                            elif 0.01 < self.velocity <= 0.1:
                                self.retrieval_frequency = 'medium'
                            else:
                                self.retrieval_frequency = 'low'
                        

                            # Get historical popularity data
                            historical_popularity = self.get_historical_popularity()
                            
                            # If there are not enough data points to determine a trend, default to 'stable'
                            if len(historical_popularity) < 3:
                                self.trend = 'stable'
                                return

                            # Use the previously calculated mean popularity
                            mean_popularity = self.mean_popularity
                            
                            # Compare current popularity with mean popularity
                            if self.current_popularity > mean_popularity:
                                # Determine trend direction based on velocity
                                if self.velocity > 0:
                                    self.trend = 'up'  # Upward trend
                                elif self.velocity == 0:
                                    self.trend = 'stable'  # No significant change
                                else:
                                    self.trend = 'down'  # Current popularity is decreasing

                            elif self.current_popularity < mean_popularity:
                                # Determine trend direction based on velocity
                                if self.velocity < 0:
                                    self.trend = 'down'  # Downward trend
                                elif self.velocity == 0:
                                    self.trend = 'stable'  # No significant change
                                else:
                                    self.trend = 'up'  # Current popularity is improving

                            else:
                                self.trend = 'stable'  # Current popularity equals mean
                        
                        

Feature Importance

To determine the significance of each feature in predicting trends, I calculate feature importance using various machine learning algorithms. These include RandomForest, Extra Trees Classifier, HistGradientBoosting, and others. After training the models, I evaluate which features have the greatest impact on the predictions. The combined insights from both Spotify attributes and calculated variables allow us to focus on the most predictive features. The models being utilized will be discussed in further detail in the next section:

• RandomForest
• HistGradientBoosting
• LogisticRegression
• SVM
• LDA
• ExtraTrees
• KNN

The above values are saved along with the model data to ensure that we can trace which features played the most critical role in the predictions.

3. Model Selection and Breakdown

This section provides a comprehensive overview of the models employed in the trend analysis, along with a detailed breakdown of each model's performance, strengths, and weaknesses. We will analyze the effectiveness of various algorithms in capturing the nuances of musical trends and making predictions based on historical data.

Random Forest

Strengths: RandomForest is a versatile and robust model known for its ability to handle both classification and regression problems with high accuracy. It can automatically handle missing data and capture complex relationships between variables.

Weaknesses: One of the downsides is that it can be slower to train, especially with large datasets, and might not perform as well as other algorithms for high-dimensional data.

HistGradientBoosting

Strengths: This algorithm is particularly strong in handling large datasets and complex data relationships, often outperforming RandomForest for smaller datasets by refining the predictive power with gradient boosting.

Weaknesses: Gradient boosting can be prone to overfitting, especially if hyperparameters are not tuned properly. It also requires more training time.

Logistic Regression

Strengths: LogisticRegression is a fast, simple, and interpretable model, especially useful when the relationship between the features and the target variable is approximately linear. It works well with smaller datasets and binary classification tasks.

Weaknesses: Its simplicity limits its performance with non-linear data, and it may underperform if the features are not well-scaled.

SVM (Support Vector Machine)

Strengths: SVM is highly effective in high-dimensional spaces and can handle both classification and regression. It is powerful for datasets with clear margin separation.

Weaknesses: SVM can be computationally intensive, especially with larger datasets, and performance can vary significantly based on the choice of kernel.

LDA (Linear Discriminant Analysis)

Strengths: LDA is an excellent model for classification, particularly when the data follows a Gaussian distribution. It reduces the dimensionality while maximizing class separability.

Weaknesses: LDA assumes linear separability and equal covariance matrices across classes, which limits its performance on non-linear or complex datasets.

Extra Trees Classifier

Strengths: Similar to RandomForest, Extra Trees is efficient for large datasets and often performs better by introducing randomness in feature selection and splitting.

Weaknesses: It shares similar downsides with RandomForest, such as sensitivity to noisy data and increased computational time.

K-Nearest Neighbors (KNN)

Strengths: KNN is intuitive, simple, and effective, particularly for smaller datasets. It can capture local patterns and relationships in the data.

Weaknesses: It can struggle with larger datasets, as it is computationally expensive, and performance can degrade with irrelevant or redundant features.

Technical Insights

This section explains key technical components used in the project:

• Data Preprocessing: The dataset is processed using SimpleImputer for handling missing values and StandardScaler for scaling numerical features, ensuring models like LogisticRegression, SVM, and KNN perform optimally.
• Model Training: We utilize GridSearchCV to fine-tune hyperparameters for each model. For instance, the parameter grids used for RandomForest and HistGradientBoosting include varying estimators, learning rates, and depth configurations to ensure the best model fit for the data.
• Feature Importance: For models supporting feature importance (e.g., RandomForest, Extra Trees), we extract the importance values directly from the model attributes. For others, such as SVM and HistGradientBoosting, permutation importance is calculated using permutation_importance from Scikit-learn to assess feature relevance.
• Model Saving: The joblib library is used to serialize and save trained models, imputation strategies, and feature names for reproducibility.
• Performance Metrics: The evaluation includes accuracy, classification reports, and confusion matrices. These metrics allow us to assess the performance and identify areas for model improvement.