Loxessis in the Security-Surveillance Technology Industry

Loxess

loxess

Loss of Amazon Rainforest Resilience: Author Contributions

Loss of Amazon Rainforest Resilience: Materials and Methods

Introduction Time series data, which consists of data points arranged in chronological order, is vital in diverse sectors such as finance, healthcare, and meteorology. The art of feature engineering, where new features are derived from existing data, is a crucial aspect of developing precise and dependable predictive models. In this article, we will discuss some essential base tips for feature engineering with time series data to help you harness the full potential of your data and improve the performance of your machine learning models. We will delve into date and time features, domain-specific features, lag features, rolling and expanding window functions, exponential smoothing, and seasonal decomposition. By mastering these techniques, you will be better equipped to uncover hidden patterns, trends, and relationships within your time series data and enhance your model's ability to make accurate predictions. Tips 1. Date and Time Features and Domain-Specific Features Time series data often comes with timestamps that provide valuable information about the data. Extracting date and time features such as day, month, year, day of the week, and hour can help improve the model's performance by capturing patterns and trends related to these temporal components. Seasonality, holidays, and weekends are examples of cyclical patterns that can be identified using date and time features. Incorporating domain-specific features can significantly enhance the performance of a machine-learning model. Domain-specific features are derived from expert knowledge in the relevant field and can provide valuable information that is not present in the raw time series data. For example, in finance, the volatility index or other economic indicators can be used as additional features to improve the prediction of stock prices. Here is an example of my own function that was widely used across quant hedge fund practices: def extract_date_features(data):
    data['Year'] = data.index.year
    data['Month'] = data.index.month
    data['Day'] = data.index.day
    data['Weekday'] = data.index.weekday
    data['Day_of_year'] = data.index.dayofyear
    data['Week_of_year'] = data.index.isocalendar().week
    data['Quarter'] = data.index.quarter
    data['Is_month_start'] = data.index.is_month_start
    data['Is_month_end'] = data.index.is_month_end
    data['Is_quarter_start'] = data.index.is_quarter_start
    data['Is_quarter_end'] = data.index.is_quarter_end
    data['Is_year_start'] = data.index.is_year_start
    data['Is_year_end'] = data.index.is_year_end
    data['Days_in_month'] = data.index.days_in_month
    data['Is_leap_year'] = data.index.is_leap_year
    data['Elapsed_days'] = (data.index - data.index.min()).days
    data['Weekday_name'] = data.index.day_name()
    data['Month_name'] = data.index.month_name()
    data['Is_weekend'] = data['Weekday'].apply(lambda x: x >= 5)
    data['Is_weekday'] = ~data['Is_weekend']
    data['Days_till_month_end'] = data['Days_in_month'] - data['Day']
    data['Days_since_month_start'] = data['Day'] - 1
    data['Week_of_month'] = (data['Day'] - 1) // 7 + 1
    data['Weekday_of_month'] = (data['Day'] - 1) % 7 + 1
    data['Days_to_next_holiday'] = data.index.to_series().apply(lambda x: (x + pd.DateOffset(days=1)).to_period('D').start_time)
    data['Days_since_last_holiday'] = data.index.to_series().apply(lambda x: (x - pd.DateOffset(days=1)).to_period('D').end_time)
    data['Business_days_in_month'] = data.index.to_series().apply(lambda x: np.busday_count(x.replace(day=1), x.replace(day=x.days_in_month) + pd.DateOffset(days=1)))
    data['Business_day_of_month'] = data.index.to_series().apply(lambda x: np.busday_count(x.replace(day=1), x))
    data['Days_since_first_day_of_year'] = data['Day_of_year'] - 1
    data['Days_remaining_in_year'] = (data.index + pd.offsets.YearEnd(0)).dayofyear - data['Day_of_year']
    
    return data

data = extract_date_features(data)
print(data.head()) 2. Lag Features Lag features refer to the values of previous time steps in the time series. They can help capture the autocorrelation present in the data, which is the relationship between the current value and its past values. Adding lag features can improve a model's performance by allowing it to learn patterns from the past to predict future values. You can create lag features by shifting the original time series data by a specific number of periods, often called the "lag order." import pandas as pd

def create_lag_features(data, n_lags):
    data_frame = pd.DataFrame(data, columns=['value'])
    for i in range(1, n_lags+1):
        data_frame[f'lag_{i}'] = data_frame['value'].shift(i)
    return data_frame

time_series = [100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200]
n_lags = 3
lag_features = create_lag_features(time_series, n_lags)

#Result:

   value  lag_1  lag_2  lag_3
0    100    NaN    NaN    NaN
1    110  100.0    NaN    NaN
2    120  110.0  100.0    NaN
3    130  120.0  110.0  100.0
4    140  130.0  120.0  110.0
5    150  140.0  130.0  120.0
6    160  150.0  140.0  130.0
7    170  160.0  150.0  140.0
8    180  170.0  160.0  150.0
9    190  180.0  170.0  160.0
10   200  190.0  180.0  170.0 3. Rolling and Expanding Windows Functions Rolling window and expanding window statistics involve computing summary statistics such as mean, median, standard deviation, and maximum or minimum values over a moving or progressively larger window of fixed size. These features help capture the local trends, fluctuations, and overall behavior in the time series data, allowing the model to learn from the temporal dynamics. Rolling window statistics can be particularly useful when dealing with noisy data or when the time series exhibits non-stationarity. Expanding windows, on the other hand, can provide insights into the cumulative effects or long-term trends in the data. Both rolling and expanding window statistics can be easily implemented using the rolling and expanding functions available in the pandas library. Code example: # Apply a rolling window
window_size = 3
time_series['Rolling_Mean'] = time_series['Data'].rolling(window=window_size).mean()
time_series['Rolling_Std'] = time_series['Data'].rolling(window=window_size).std()

# Apply an expanding window
time_series['Expanding_Mean'] = time_series['Data'].expanding().mean()
time_series['Expanding_Std'] = time_series['Data'].expanding().std()

# Visualize the results
fig, ax = plt.subplots(figsize=(10, 5))
ax.plot(time_series['Date'], time_series['Data'], label='Original Data', marker='o')
ax.plot(time_series['Date'], time_series['Rolling_Mean'], label='Rolling Mean', marker='o')
ax.plot(time_series['Date'], time_series['Expanding_Mean'], label='Expanding Mean', marker='o')
ax.legend(loc='best')
ax.set_title('Rolling and Expanding Window Example')
plt.show() 4. Exponential Smoothing Exponential smoothing is a technique that assigns exponentially decreasing weights to past observations, placing more emphasis on recent observations. This approach can be useful for reducing noise and capturing trends in the time series data. There are various forms of exponential smoothing such as simple exponential smoothing, double exponential smoothing (Holt's method), and triple exponential smoothing (Holt-Winters method), which can be employed based on the characteristics of the time series data. Simple Exponential smoothing is another method used to smooth time series data. It assigns exponentially decreasing weights to past observations, giving more importance to recent data points. The formula for exponential smoothing is: Where ES(t) is the exponentially smoothed value at time t, X_t is the data point at time t, and α is the smoothing factor (0 < α < 1). Simple Python example: def exponential_smoothing(data, alpha):
    es = [data[0]]
    for t in range(1, len(data)):
        es.append(alpha * data[t] + (1 - alpha) * es[t - 1])
        return es Holt-Winters is a forecasting method that extends the simple Exponential Smoothing method, accounting for seasonality and trend. The method consists of three components: level (smoothing of the series), trend (smoothing of the trend), and seasonality (seasonal component). The method comes in two flavors: additive and multiplicative. Holt-Winters code example: from statsforecast.models import HoltWinters
from statsforecast.utils import AirPassengers as ap


model = HoltWinters(season_length=12, error_type='A')
model = model.fit(y=ap)
y_hat_dict = model.predict(h=4)
y_hat_dict 5. Seasonal Decomposition Seasonal decomposition is a technique to separate a time series into its trend, seasonal, and residual components. The most common method for seasonal decomposition is using the STL (Seasonal and Trend decomposition using Loess) method. Code example for seasonal decomposition using traditional library: statsmodels import statsmodels.api as sm
import matplotlib.pyplot as plt

data = np.array([...])
# Input time series data
seasonal_period = 12
result = sm.tsa.seasonal_decompose(data, period=seasonal_period)
trend = result.trend
seasonal = result.seasonal
residual = result.resid
result.plot()
plt.show() An additional code example for seasonal decomposition using library: statsforecast from statsforecast import StatsForecast
from statsforecast.models import MSTL, AutoARIMA
import matplotlib.pyplot as plt

models = [
    MSTL(
        season_length=[12 * 7],  # seasonalities of the time series
        trend_forecaster=AutoARIMA(),  # model used to forecast trend
    )
]

sf = StatsForecast(
    models=models,  # model used to fit each time series
    freq="D",  # frequency of the data
)
sf = sf.fit(data)
test = sf.fitted_[0, 0].model_
fig, ax = plt.subplots(1, 1, figsize=(10, 8))
test.plot(ax=ax, subplots=True, grid=True)
plt.tight_layout()
plt.show() Conclusion Time series data is indispensable for various industries, and feature engineering plays a pivotal role in developing accurate and reliable predictive models. In this article, we have discussed essential techniques for feature engineering with time series data, such as extracting date and time features, incorporating domain-specific features, generating lag features, using rolling and expanding window functions, implementing exponential smoothing, and performing seasonal decomposition. By mastering these approaches, you will be better equipped to harness the full potential of your time series data and enhance your model's ability to make accurate predictions. Remember that every time series dataset is unique, and it's crucial to tailor your feature engineering strategies to your specific problem domain. Experiment with different techniques, combinations, and parameters to uncover hidden patterns and relationships in your data and create models that are better suited to address the challenges of your particular domain. By doing so, you'll be well on your way to unlocking the full potential of your time series data and creating more robust and accurate predictive models.

Must-Know Base Tips for Feature Engineering With Time Series Data

Geological Distribution Part II

This by David, Linh, Ellen, Hang, Utsav Jaiswal, Bruno Molteni and Sidra Ijaz occurred in hackernoon's official #only-at-hackernoon channel. Slogging thread David Sep 1, 2021, 5:50 PM The secret to being a good is steering into your adhd hackernoon employee 😂 6 💚 5 :dead-inside: 4 Linh Sep 1, 2021, 5:51 PM Hang Utsav Jaiswal to debate the pros and cons Ellen Sep 1, 2021, 5:52 PM But organize your email tho Hang Sep 1, 2021, 5:53 PM You get your hands on all the task but we got nothing done :dead-inside: Utsav Jaiswal Sep 1, 2021, 8:59 PM Classic ADHD tangent - Wait!! Does that mean I'm a good employee? Nice!! 💚 2 :smiling_face_with_tear: 1 Linh Sep 1, 2021, 9:05 PM YOU UNDERSTAND EVERYTHING that’s going on at the company at the same time you still don’t understand: 1. blockchain 2. metaverse 3.anything that has anything to do with coding. :savage-joy: 3 Utsav Jaiswal Sep 1, 2021, 9:07 PM Blockchains are like the spreadsheets that do not have a delete key :exploding_head: 2 Hang Sep 1, 2021, 9:14 PM classic ADHD case is when you are in a meeting and you reply to Slack threads and answer emails at the same time :savage-joy: :savage-joy: 2 Linh Sep 1, 2021, 9:24 PM CLASSIC ADHD: WE MENTIONED HALLUCINOGENIC MUSHROOMS out of nowhere in a brand-dashboard meeting :woman-tipping-hand: Linh Sep 1, 2021, 9:24 PM (the meeting is still going on, btw) :savage-joy: 1 Utsav Jaiswal Sep 1, 2021, 9:26 PM attention deficit on this thread is over 9000 :savage-joy: Ellen Sep 1, 2021, 9:27 PM Which meeting was this? Sounds like I should have been there. Hang Sep 1, 2021, 9:27 PM It turns out we all have short attention span Hang Sep 1, 2021, 9:28 PM And today we learn Bruno looks like Sidra’s younger brother, also during the meeting :savage-joy: 😂 3 Ellen Sep 1, 2021, 9:28 PM The first step to admitting you have a 'problem,' is organizing your email. 🤔 2 Linh Sep 1, 2021, 9:33 PM Bruno Molteni LOL Bruno Molteni Sep 1, 2021, 9:35 PM Sidra Ijaz please send pic ... now I want to know! Sidra Ijaz Sep 1, 2021, 9:35 PM Yes that picture would be perfect for this thread 😂 David Sep 1, 2021, 9:36 PM OH: "David, are you trying to explain the metaverse to me?" Sidra Ijaz Sep 1, 2021, 9:36 PM Bruno Molteni’s Pakistani doppelgänger :party_parrot: 2 Linh Sep 1, 2021, 9:38 PM here’s how that convo went: David: I think we can position HackerNoon Content Management System not as a digital site but a digital economy. You have readers, writers, sponsors, each can do their own things or interact with each other. Linh: did you just try to explain the metaverse to me or have i been on too many meetings? 😂 2 David Sep 1, 2021, 9:40 PM This is Chad: https://giphy.com/gifs/pixelated-hackernoon-face-x5eCE2eTD6J0mr8OFZ 😂 1 David Sep 1, 2021, 9:41 PM This is Chad in a Haunted Mansion: David Sep 1, 2021, 9:43 PM my emoji game :awwyeah: 1 David Sep 1, 2021, 9:44 PM When you discover an undocumented technology, you should document it to help the next person looking for it. Hang Sep 1, 2021, 9:49 PM Do ADHD symptoms include reacting with irrelevant emojis to a post? asking for a fren 🤖 2 👀 1 :savage-joy: 1 :fireball: 1 David Sep 1, 2021, 9:53 PM The only thing that rhymes with orange is , which is a very rare alternative to the fern-like . sporange https://www.lexico.com/definition/sporangium :orange_heart: 1 🍊 1 😆 1 Hang Sep 1, 2021, 9:57 PM Now that you mention it I do want an orange David Sep 1, 2021, 10:01 PM If you find yourself in the area, buy some Redlands oranges. https://www.yelp.com/search?find_desc=Fruit+Stands&find_loc=Redlands%2C+CA Sidra Ijaz Sep 2, 2021, 6:17 AM “Blessed are the forgetful, for they get the better even of their blunders.” - Nietzsche 💯 2 ❤ 2 Ellen Sep 2, 2021, 6:25 AM You did not just quote Nietzsche, I'm dead. :smiling_face_with_tear: 4 ❤ 3 💚 2 Linh Sep 2, 2021, 1:41 PM Omg you know you are next level smart when you just quote Nietzsche but pleaseeeeeee watch envy by contrapoint she lovesszzzzz him 😆 💯 1 Sidra Ijaz Sep 2, 2021, 1:43 PM I have written that in my notes! 🙂 I'll watch that video 😄 Sidra Ijaz Sep 2, 2021, 2:49 PM lol, she has butchered Nietzsche. AND SHE IS MIND BLOWING. V well-read. Thank you for recommending her content Linh 🙂 :party_parrot: 1 Linh Sep 2, 2021, 6:03 PM David wanna slog this? David Sep 2, 2021, 6:09 PM https://www.youtube.com/watch?v=gJLIiF15wjQ

Slogging Definition: Publish Remote Watercooler Moments

The “Maybe Just a Quick One” series title is inspired by my most common reply to “Fancy a drink?”, which, may or may not end up in a long night. Likewise, these posts are intended to be short but I get carried away sometimes, so, apologies in advance. What is Exploratory Data Analysis and why is it important? So we know what data is and we know what analysis is. But what is the meaning of “exploratory” in a data science context? What kind of conclusions are we trying to reach? Well, there are various reasons that make this step a necessity in a data science project's lifecycle, helping us to: Make a good judgement on the quality of our data. Identify any missing values, outliers, possible differences in measurement units etc. Take a closer look at the characteristics of the variables like their types, distributions, variance as well as the correlations between them. Summarise the data, providing an “at a glance” way to understand it. And this works beneficially not just for the data scientist or the developer but any stakeholder involved in the project . One of the most common ways to achieve this is by using graphs and plots, in other words, to visualize the data. What is Autoviz? is a Python library that can massively speed up the visualization of our data, making it fully automated. Let’s jump straight into coding. I am a firm believer in learning by doing. Autoviz First things first. We will create a new Conda environment and install the necessary packages: conda create -n autoviz  python= conda activate autoviz
python -m pip install autoviz
conda install  scikit-learn 3.8 The reason I included scikit-learn is to use some of its datasets to demonstrate the use of Autoviz. You can of course download the dataset from other sources and skip this step. Now, create a new notebook and start by importing the packages. I will use the infamous Boston House Prices dataset first. autoviz.AutoViz_Class AutoViz_Class sklearn.datasets load_boston,load_iris pandas pd

boston = load_boston()
df_boston = pd.DataFrame(data=boston.data, columns=boston.feature_names)
df_boston[ ] = boston.target from import from import import as "btarget" We can now instantiate Autoviz and let it do its magic: AV = AutoViz_Class()
filename = sep = dft = AV.AutoViz(
    filename,
    sep= ,
    depVar= ,
    dfte=df_boston,
    header= ,
    verbose= ,
    lowess=False,
    chart_format= ,
    max_rows_analyzed= ,
    max_cols_analyzed= ,
) "" "," "," "btarget" 0 2 "svg" 150000 30 As you might have noticed, there are some arguments passed to AutoViz, but what do they mean? Let's see what the documentation says: - Make sure that you give filename as empty string ("") if there is no filename associated with this data and you want to use a dataframe, then use dfte to give the name of the dataframe. Otherwise, fill in the file name and leave dfte as empty string. Only one of these two is needed to load the data set. filename - this is the separator in the file. It can be comma, semi-colon or tab or any value that you see in your file that separates each column. sep - target variable in your dataset. You can leave it as empty string if you don't have a target variable in your data. depVar - this is the input dataframe in case you want to load a pandas dataframe to plot charts. In that case, leave filename as an empty string. dfte - the row number of the header row in your file. If it is the first row, then this must be zero. header - it has 3 acceptable values: 0, 1 or 2. With , you get all charts but limited info. With you get all charts and more info. With , you will not see any charts but they will be quietly generated and save in your local current directory under the AutoViz_Plots directory which will be created. Make sure you delete this folder periodically, otherwise, you will have lots of charts saved here if you used verbose= option a lot. verbose zero 1 2 2 - this option is very nice for small datasets where you can see regression lines for each pair of continuous variable against the target variable. Don't use this for large data sets (that is over 100,000 rows) lowess - this can be SVG, PNG or JPG. You will get charts generated and saved in this format if you used verbose=2 option. Very useful for generating charts and using them later. chart_format - limits the max number of rows that is used to display charts. If you have a very large data set with millions of rows, then use this option to limit the amount of time it takes to generate charts. We will take a statistically valid sample. max_rows_analyzed - limits the number of continuous vars that can be analyzed max_cols_analyzed Wait...that was it? Yes. It is that simple. Using 2 as the level had the charts generated in the folder. Let's  take a look at some of them: verbose AutoViz_Plots Violin Plots Scatter Plots Heatmaps and more. You get the gist of it. (I hear you say) That is cool. I won't need to do any data vizualisation myself anymore! Not quite. I believe that we need the best of both worlds. Having an automated data visualization tool like Autoviz to quickly generate some graphs for your data is a great first step. It can very quickly give you a good summary of it. However, you might need to dig deeper and create some plots yourself, depending on the task. Further reading: Autoviz home page

Easy Data Visualization with AutoViz [Maybe Just a Quick One]

Want to make money in your sleep? Automated trading may be the perfect fit for you! This discipline of trading involves breaking your strategy down into a series of entry and exit rules using technical indicators and live market data. Once this is done, a program will monitor live price data from crypto exchanges and make trades based off your defined rules. Fortunately, there are many that make this entire process much easier than it sounds! professional tools In this article, we explore the top 5 reasons why you should consider automating your crypto trading strategy. 1. Crypto Trading Bots Remove The Emotion From Your Trading Whether you realize it or not, your emotions play a huge role in your trading decisions. How many times have you decided to lower a stop-loss because you “knew” the asset would bounce back. Or how many times have you held onto a position past your profit target and ended up with less? With an automated trading system, you define your buy signals, your stop losses, and your profit targets all beforehand. The trading bot will execute these set of rules for you, taking your emotions completely out of the picture. Doing this will prevent you from getting too cocky on a win streak and from trying to overcompensate your trades while on a losing streak . Over time, this consistency can go a very long way to increasing your bottom line. 2. You Can Test A Strategy’s Profitability Against Historical Data Almost all trading bots and automated trading systems are built with a backtester. A backtester is a tool that will simulate the performance of a strategy against years of historical data to determine its profitability and risk. After running a backtest, the program will output helpful statistics to determine the profitability and effectiveness of your strategy. Some of the most important metrics to consider for your trading strategies: Number of Trades Profit vs Benchmark (Such as buying and holding Bitcoin) Win / Loss Ratio Max Consecutive Losses Max Drawdown (Single largest drop in portfolio value) When backtesting your strategy, it is always best to minimize risk and drawdown as much as possible before you trade it live. Failing to do this, can result in even if your backtesting results determined it to be a profitable trading strategy. unexpected losses 3. Trading Bots Can Run Non-Stop Across Many Markets One of the most obvious benefits of using a crypto trading bot is its ability to run . Since crypto exchanges don’t shut down, your potential to make profit increases greatly! In addition to this, most trading bots allow you to configure and execute your strategies across multiple trading pairs. 24/7/365 4. Improved Trading Speed & Order Execution Using a crypto trading bot will allow you to time and execute your entry/exit points much more efficiently than trading by hand. Without the need to be constantly logged in across multiple exchanges, your can place orders much faster and get more profit out of each and every trade. 5. Detailed Trade Reports & Live Notifications Keeping track of trades and calculating your gains can be a huge pain when doing it by hand. Almost all trading bots come with a helpful trading dashboard that summarizes profits/losess for each of your strategies. Most also integrate very well with popular messaging platforms (like Telegram) and will ping updates whenever your trading bot decides to put in an order. crypto tools Crypto trading bots can be very profitable when they are configured properly and backtested by the trader. Before jumping in, it is important that you know how a strategy is expected to behave in a bull, bear, and sideways market. Backtesting your strategy is one of the best way to do this and should be done before taking it live. If you would like to start automating your trading strategies, take a look at some of the built specifically for crypto traders. professional trading bots . Easily report your cryptocurrency capital gains with CryptoTrader .Tax Originally published at www.cryptotrader.tax on April 23, 2018.

Stories, News & Mentions about LOXESS

Loxess’s Stories on HackerNoon