Bitcoin whales step up buying as a BTC price recovery sees $62,000 return on U.S. jobs data.

Bitcoin price springs 5% to $62K as US jobs data boosts rate cut bets

After a government employment report was published, showing unemployment is higher than expected, Bitcoin leaped over $61k.

Bitcoin Price Leaps Over $61K Following US Unemployment Data - Decrypt

Wall Street stocks rose early Friday, greeting Apple&#x27;s mammoth shares buyback and signs of slowing in the US jobs market that could prompt a faster shift in monetary policy.

US Stocks Rally On Jobs Data, Apple Buyback | Barron's

Goldman Sachs raised its stock price target to $268 after First Solar reported another stronger quarter, suggesting 50% upside for shares.

This solar stock is poised for 50% upside as demand strengthens on data center surge, Goldman says

Connecting decision makers to a dynamic network of information, people and ideas, Bloomberg quickly and accurately delivers business and financial information, news and insight around the world · Americas+1 212 318 2000

Hacking Democracy: UK, US Sanction China For Stealing Voter Data - Bloomberg

We talk AI, data centers, and startups with John Yue, the CEO and Co-Founder of Inference.ai

Interview: The truth about business AI, data centers, and why startups need support too | TechRadar

But home ownership still out of reach for many because prices remain high

England’s housing has become more affordable, new data shows

Exclusive ‘exit data’ obtained by Byline Times suggests the force has a long way to go in its claimed attempts to &#x27;address valid concerns&#x27; about life as a non-white or female employee on the force.

Revealed: Met Police 'Exit Data' Shows Just How Bad it is to Work there for Some Staff – Byline Times

GSTR-1 ERROR RESOLUTION: The Table 8 has been freshly designed this time to bring in changes of declaration of e-commerce data. Open Table 8 of GSTR 1 and save it. After saving, then generate summary. Now the error will be resolved. Source #GST Portal

Resolve GSTR-1 Error: Update and Save E-commerce Data in Table 8 for Error-Free Summary Generation | A2Z Taxcorp LLP

Bruno Aziza shares 2024 tech and business predictions, offers insights on AI strategy and discusses how you might want to measure your life.

This week in data: Why metrics don't matter | VentureBeat

YData

ydata

In this blog post, we will be covering how you can combine and leverage the open-source streaming solution, , with , to improve the quality of your streaming flows. Buckle up! bytewax ydata-profiling Stream processing enables real-time analysis of data in-flight and before storage and can be or . stateful stateless is used for recommendations, pattern detection, or complex event processing, where the history of what has happened is required for the processing (windows, joining by a key, etc.). Stateful stream processing real-time is used for transformation that doesn’t require knowledge of other data points in the stream like masking an email or converting a type. Stateless stream processing inline Overall, data streams are widely used in the industry and can be found applied to use cases such as , , or . fraud detection patient monitoring event predictive maintenance One Crucial Aspect That All Data Streams Must Consider Is the Quality of the Data Unlike traditional models where data quality is usually assessed during the creation of the data warehouse or dashboard solution, . streaming data requires continuous monitoring It is essential to maintain data quality throughout the entire process, from collection to feeding downstream applications. After all, the cost of bad data quality can be high for organizations: “The cost of bad data is an astonishing 15% to 25% of revenue for most companies. (…) Two-thirds of these costs can be eliminated by getting in front on data quality.” — Thomas C. Redman, author of “Getting in Front on Data Quality” Throughout this article, we will show you how you can combine with to profile and improve the quality of your streaming flows! bytewa ydata-profiling Stream Processing for Data Professionals With Bytewax is an OSS stream processing framework designed specifically for Python developers. Bytewax It allows users to with capabilities similar to Flink, Spark, and Kafka Streams while providing a friendly and familiar interface and build streaming data pipelines and real-time applications 100% compatibility with the Python ecosystem. Using built-in or existing Python libraries, (Kafka, RedPanda, WebSocket, etc.) and out to various downstream systems (Kafka, parquet files, data lakes, etc.). connectors you can connect to real-time and streaming data sources write transformed data For the transformations, Bytewax with , , and methods and comes with familiar features such as recovery and scalability. facilitates stateful and stateless transformations map windowing aggregation Bytewax and is p . facilitates a Python-first and data-centric experience to data streams urposely built for data engineers and data scientists It allows users to and create customizations necessary to meet their needs without having to learn and maintain JVM-based streaming platforms like Spark or Flink. build streaming data pipelines and real-time applications Bytewax is well suited for many use cases, namely, , , , and more. Embedding Pipelines For Generative AI Handling Missing Values in Data Streams Using Language Models in a Streaming Context to Understand Financial Markets For use case inspiration and more information like documentation, tutorials, and guides, feel free to check . the bytewax website Why Data Profiling for Data Streams? and refers to the step of : its structure, behavior, and quality. Data Profiling is key to a successful start of any machine learning task thoroughly understanding our data In a nutshell, involves analyzing aspects related to the data’s format and basic descriptors (e.g., number of samples, number/types of features, duplicate values), its (such as the presence of missing data or imbalanced features), and other complicating factors that may arise during data collection or processing (e.g., erroneous values or inconsistent features). data profiling intrinsic characteristics , where circumstances might change fast and may require immediate action (e.g., healthcare monitoring, stock values, air quality policies). Ensuring high data quality standards is crucial for all domains and organizations, but is especially relevant for domains operating with domains outputting continuous data For many domains, data profiling is used from an exploratory data analysis lens, considering historical data stored in databases. On the contrary, for data streams, , where data needs to be checked at different time frames or stages of the process. data profiling becomes essential for validation and quality control continuously along the stream By embedding an , we can on the current state of our data and be alerted for any potentially critical issues — whether they are related to (e.g., corrupted values or changing formats), or to (e.g., data drifts, deviation from business rules and outcomes). automated profiling into our data flows immediately get feedback data consistency and integrity events happening in short periods of time In real-world domains — — automated profiling might save us from multiple brain puzzles and systems needing to be taken out of production! where you just know Murphy’s law is bound to strike and “everything can definitely go wrong” In what concerns data profiling, has consistently been a , either for or data. And no wonder why — ydata-profiling crowd favorite tabular time-series it’s one line of code for an extensive set of analysis and insights. Complex and time-draining operations are done under the hood: ydata-profiling and depending on the feature types (either numeric or categorical), it that are shown in the profiling report. automatically detects the feature types comprised in the data, adjusts the summary statistics and visualizations Fostering a , the package also , focusing on their pairwise and , and provides a thorough , from or values to and features. data-centric analysis highlights the existing relationships between features interactions correlations evaluation of data quality alerts duplicate constant skewed imbalanced It’s really a — with minimal effort. 360º view of the quality of our data Putting It All Together: bytewax and ydata-profiling Before starting the project, we need to first set our Python dependencies and configure our data source. First, let’s install the and packages ( bytewax ydata-profiling You might want to use a virtual environment for this — if you need some extra guidance!) check these instructions pip install bytewax==0.16.2 ydata-profiling==4.3.1 Then, we’ll upload the (License — CC0: Public Domain), which contains several measurements of from different IoT devices: Environmental Sensor Telemetry Dataset temperature, humidity, carbon monoxide liquid petroleum gas, smoke, light, and motion wget https://raw.githubusercontent.com/bytewax/air-quality-sensor/main/data/iot_telemetry_data_1000 , and the input would look like what we expect in a streaming platform . In this article, and create a dataflow using bytewax. In a production environment, these measurements would be continuously generated by each device such as Kafka to simulate the context we would find with streaming data, we will read the data from the CSV file one line at a time (As a quick side note, a dataflow is essentially a data pipeline that can be described as a directed acyclic graph — DAG) First, let’s make some : necessary imports from datetime import datetime, timedelta, timezone

from bytewax.dataflow import Dataflow
from bytewax.connectors.stdio import StdOutput
from bytewax.connectors.files import CSVInput
from bytewax.testing import run_main Then, we define our dataflow object. Afterward, we will use a stateless map method where we pass in a function to convert the string to a DateTime object and restructure the data to the format (device_id, data). The map method will make the change to each data point in a stateless way. The reason we have modified the shape of our data is so that we can easily group the data in the next steps to profile data for each device separately rather than for all of the devices simultaneously. flow = Dataflow()
flow.input("simulated_stream", CSVInput("/content/iot_telemetry_data_1000"))

# parse timestamp
def parse_time(reading_data):
    reading_data["ts"] = datetime.fromtimestamp(float(reading_data["ts"]), timezone.utc)
    return reading_data

flow.map(parse_time)


# remap format to tuple (device_id, reading_data)
flow.map(lambda reading_data: (reading_data['device'], reading_data)) Now, we will take advantage of the stateful capabilities of to gather data for each device over the duration of time that we have defined. expects a snapshot of the data over time, which makes the window operator the perfect method to use to do this. bytewax ydata-profiling In , we are able to produce summarizing statistics for a dataframe that is specified for a particular context. For instance, in our example, we can produce snapshots of data referring to each IoT device or to particular time frames: ydata-profiling from bytewax.window import EventClockConfig, TumblingWindow

# This is the accumulator function, and outputs a list of readings
def acc_values(acc, reading):
    acc.append(reading)
    return acc


# This function instructs the event clock on how to retrieve the
# event's datetime from the input.
def get_time(reading):
    return reading["ts"]

  
# Configure the `fold_window` operator to use the event time.
cc = EventClockConfig(get_time, wait_for_system_duration=timedelta(seconds=30))

# And a tumbling window
align_to = datetime(2020, 1, 1, tzinfo=timezone.utc)
wc = TumblingWindow(align_to=align_to, length=timedelta(hours=1))

flow.fold_window("running_average", cc, wc, list, acc_values)

flow.inspect(print) After the snapshots are defined, leveraging is as simple as calling the for each of the dataframes we would like to analyze: ydata-profiling ProfileReport import pandas as pd
from ydata_profiling import ProfileReport

def profile(device_id__readings):
    print(device_id__readings)
    device_id, readings = device_id__readings
    start_time = readings[0]['ts'].replace(minute=0, second=0, microsecond=0).strftime('%Y-%m-%d %H:%M:%S')
    df = pd.DataFrame(readings)
    profile = ProfileReport(
        df,
        tsmode=True,
        sortby="ts",
        title=f"Sensor Readings - device: {device_id}"
    )

    profile.to_file(f"Ts_Profile_{device_id}-{start_time}.html")
    return f"device {device_id} profiled at hour {start_time}"


flow.map(profile) In this example, we are writing the images out to local files as part of a function in a map method. These could be reported out via a messaging tool or we could save them to some remote storage in the future. Once the profile is complete, the dataflow expects some output so we can use the built-in to print the device that was profiled and the time it was profiled at that was passed out of the profile function in the map step: StdOutput flow.output("out", StdOutput()) There are multiple ways to execute Bytewax dataflows. In this example, we use the same local machine, but Bytewax can also run on multiple Python processes, across multiple hosts, in a , using a , and . Docker container Kubernetes cluster more In this article, we’ll continue with a local setup, but we encourage you to check our helper tool which manages Kubernetes dataflow deployments once your pipeline is ready to transition to production. waxctl Assuming we are in the same directory as the file with the dataflow definition, we can run it using: python -m bytewax.run ydata-profiling-streaming:flow We can then use the profiling reports to validate the data quality, check for changes in schemas or data formats, and . compare the data characteristics between different devices or time windows In fact, we can leverage the that highlights the differences between two data profiles in a straightforward manner, making it easier for us to detect important patterns that need to be investigated or issues that have to be addressed: comparison report functionality snapshot_a_report = ProfileReport(df_a, title="Snapshot A")
snapshot_b_report = ProfileReport(df_b, title="Snapshot B")

comparison_report =snapshot_a_report(snapshot_b_report)
comparison_report.to_file("comparison_report.html") Ready to Explore Your Own Data Streams? Validating data streams is crucial to identify issues in data quality in a continuous manner and compare the state of data across distinct periods of time. For organizations in , , , and — all working with continuous streams of data — a , from quality assessment to data privacy. healthcare energy manufacturing entertainment utomated profiling is key to establishing data governance best practices This requires the analysis of snapshots of data which, as showcased in this article, can be achieved in a seamless way by combining and . bytewax ydata-profiling takes care of all the processes necessary to handle and structure data streams into snapshots, which can then be summarized and compared with through a comprehensive report of data characteristics. Bytewax ydata-profiling Being able to appropriately process and profile incoming data opens up a plethora of use cases across different domains, from the to the highlighting and mitigation of additional issues that derive from real-world activities, such as (e.g., fraud or intrusion/threats detection), , and other events that deviate from the expectations (e.g., data drifts or misalignment with business rules). correction of errors in data schemas and formats anomaly detection equipment malfunction Now, you’re all set to start exploring your data streams! Let us know what other use cases you find, and as always, feel free to drop us a line in the comments, or find us at the for further questions and suggestions! Data-Centric AI Community See you there! Acknowledgments ) and Oli Makhasoeva (Developer Relations @ ) -- developing . This article was written by Fabiana Clemente (Co-founder & CDO @ ) and Miriam Santos (Developer Relations @ ) -- developing -- and Zander Matheson (CEO & Founder @ YData YData ydata-profiling Bytewax Byetwax bytewax You may find additional information about the OSS packages in the respective documentations: & . ydata-profiling docs bytewax docs Also published here

How to Understand Your Data in Real-Time Using bytewax and ydata-profiling

Cómo entender sus datos en tiempo real usando bytewax y ydata-profiling

Comment comprendre vos données en temps réel en utilisant bytewax et ydata-profiling

बाइटवैक्स और वाईडेटा-प्रोफाइलिंग का उपयोग करके वास्तविक समय में अपने डेटा को कैसे समझें

bytewax と ydata-profiling を使用してデータをリアルタイムで理解する方法

Como entender seus dados em tempo real usando bytewax e ydata-profiling

Cách hiểu dữ liệu của bạn trong thời gian thực bằng cách sử dụng bytewax và ydata-profile

如何使用 bytewax 和 ydata-profiling 实时了解您的数据

Time-series or sequential data can be defined as any data that has time dependency. , huh, Well, a bit everywhere, from credit card transactions, my everyday routine and whereabouts to medical records, such as ECG and EEG’s. Cool but where can I find sequential data? Generate synthetic sequential data with TimeGAN Although sequential data is pretty common to be found and highly useful, there are many reasons that lead to not leverage it — from to the scarcity of its existence. In one of my previous posts, I’ve covered the ability of that preserves the utility and fidelity of a real datasets, nevertheless to generate tabular data is far more simple than generating datasets that should preserve privacy regulations Generative Adversarial Netoworks (GANs) to learn and generate new synthetic data temporal dynamics. To model successfully time-series data means that a model must, not only capture the datasets features distributions within each time-point but also, it should be able to capture the complex dynamics of those features across time. We must not forget also that each time sequence as a variable length associated. But being a challenging task, does not mean it is impossible! and have proposed, in 2019, — — that I’ll be covering with a practical example throughout this blog post. Jinsung Yoon Daniel Jarret a novel GAN architecture to model sequential data TimeGAN Time-series Generative Adversarial Networks TGAN or Time-series Generative Adversarial Networks, was proposed in 2019, as a GAN based framework that is able to generate realistic time-series data in a variety of different domains, meaning, sequential data with different observed behaviors. Different from other GAN architectures (eg. ) where we have implemented an unsupervised adversarial loss on both real and synthetic data, TimeGAN architecture introduces the concept of the model is encouraged to capture time conditional distribution within the data by using the original data as a supervision. WGAN supervised loss — Also, we can observe the introduction of an that is responsible to reduce the adversarial learning space dimensionality. embedding network block diagram and training scheme as depicted in the paper TimeGAN As mentioned above, TimeGAN is a framework to synthesize sequential data compose by 4 networks, that play distinct roles in the process of modelling the data: the expected and , but also, by a and models. generator discriminator recovery embedder Implementation with TensorFlow 2 What’s new about TimeGAN? Different from other the proposed framework is able to generate it’s training to handle a mixed-data setting, where both static and sequential data are able to be generated at the same time. GAN architectures for sequential data, (attributes) (features) Less sensitive to hyper parameters changes A more stable training process, when compared to other architectures. framework instantiated with RNNs For the purpose of this example, I’ve decided to keep it simple with a very similar architecture for all the 4 elements: A 3 layers network. But it’s possible to have this architectures change to more or less layers, and also to chose between and . TimeGAN GRU GRU LSTMs net_type== : i range(n_layers):
            model.add(GRU(units=hidden_units,
                      return_sequences= ,
                      name= )) : i range(n_layers):
            model.add(LSTM(units=hidden_units,
                      return_sequences= ,
                      name= ))

    model.add(Dense(units=output_units,
                    activation= ,
                    name= )) model : def make_net (model, n_layers, hidden_units, output_units, net_type= ) 'GRU' if 'GRU' for in True f'GRU_ ' {i + } 1 else for in True f'LSTM_ ' {i + } 1 'sigmoid' 'OUT' return Which results in the definition of each of the networks elements as per the code snippet below. self.hidden_dim=hidden_dim model = Sequential(name= )
        model.add(Input(shape=input_shape))
        model = make_net(model,
                         n_layers= ,
                         hidden_units=self.hidden_dim,
                         output_units=self.hidden_dim) model : class Supervisor (Model) : def __init__ (self, hidden_dim) : def build (self, input_shape) 'Supervisor' 2 return self.hidden_dim = hidden_dim
        self.net_type = net_type model = Sequential(name= )
        model.add(Input(shape=input_shape))
        model = make_net(model,
                         n_layers= ,
                         hidden_units=self.hidden_dim,
                         output_units=self.hidden_dim,
                         net_type=self.net_type) model self.hidden_dim = hidden_dim
        self.net_type=net_type model = Sequential(name= )
        model = make_net(model,
                         n_layers= ,
                         hidden_units=self.hidden_dim,
                         output_units= ,
                         net_type=self.net_type) model self.hidden_dim=hidden_dim
        self.n_seq=n_seq recovery = Sequential(name= )
        recovery.add(Input(shape=input_shape, name= ))
        recovery = make_net(recovery,
                            n_layers= ,
                            hidden_units=self.hidden_dim,
                            output_units=self.n_seq) recovery self.hidden_dim=hidden_dim embedder = Sequential(name= )
        embedder.add(Input(shape=input_shape, name= ))
        embedder = make_net(embedder,
                            n_layers= ,
                            hidden_units=self.hidden_dim,
                            output_units=self.hidden_dim) embedder self.hidden_dim=hidden_dim model = Sequential(name= )
        model.add(Input(shape=input_shape))
        model = make_net(model,
                         n_layers= ,
                         hidden_units=self.hidden_dim,
                         output_units=self.hidden_dim) model : class Generator (Model) : def __init__ (self, hidden_dim, net_type= ) 'GRU' : def build (self, input_shape) 'Generator' 3 return : class Discriminator (Model) : def __init__ (self, hidden_dim, net_type= ) 'GRU' : def build (self, input_shape) 'Discriminator' 3 1 return : class Recovery (Model) : def __init__ (self, hidden_dim, n_seq) return : def build (self, input_shape) 'Recovery' 'EmbeddedData' 3 return : class Embedder (Model) : def __init__ (self, hidden_dim) return : def build (self, input_shape) 'Embedder' 'Data' 3 return : class Supervisor (Model) : def __init__ (self, hidden_dim) : def build (self, input_shape) 'Supervisor' 2 return In what concerns the losses, the TimeGAN is composed by three: The , which refers to the auto-encoder (embedder & recovery), that in a nutshell compares how well was the reconstruction of the encoded data when compared to the original one. reconstruction loss The that, in a nutshell, is responsible to capture how well the generator approximates the next time step in the latent space. The , this one it’s already familiar to us, a it reflects the relation between the generator and discriminator networks (min-max game) supervised loss unsupervised loss Given the architecture choice and the defined losses we have three training phases: Training the autoencoder on the provided sequential data for optimal reconstruction; Training the supervisor using the real sequence data to capture the temporal behavior of the historical information, and finally; The combined training of four components while minimizing all the three loss functions mentioned previously. The full code detailing the training phases can be found at . ydata-synthetic The original implementation of TimeGAN can be found using TensorFlow 1 here . Synthetic stock data The data used to evaluate the synthetic data generated by the framework, refers to . The data has : Open, High, Low, Close, Adj Close and Volume. TimeGAN Google stock data 6 time dependent variables Prior to synthesize the data we must, first, ensure some preprocessing: Scale the series to a range between [0,1]. For convenience, I’ve decided to leverage scikit-learn’s MinMaxScaler; Create rolling windows — following the original paper recommendations, I’ve create rolling windows with overlapping sequences of 24 data points. ori_data = data[:: ] scaler = MinMaxScaler().fit(ori_data)
    ori_data = scaler.transform(ori_data) temp_data = [] i range( , len(ori_data) - seq_len):
        _x = ori_data[i:i + seq_len]
        temp_data.append(_x) idx = np.random.permutation(len(temp_data))
    data = [] i range(len(temp_data)):
        data.append(temp_data[idx[i]]) data # Method implemented here: https://github.com/jsyoon0823/TimeGAN/blob/master/data_loading.py # Originally used in TimeGAN research : def real_data_loading (data: np.array, seq_len) """Load and preprocess real-world datasets.
    Args:
      - data_name: Numpy array with the values from a a Dataset
      - seq_len: sequence length
    Returns:
      - data: preprocessed data.
    """ # Flip the data to make chronological data -1 # Normalize the data # Preprocess the dataset # Cut data by sequence length for in 0 # Mix the datasets (to make it similar to i.i.d) for in return Following the recommendations from the original paper, I’ve decided to train the synthesizer for 10000 iterations nevertheless, bare in mind, that these values must the optimized for each data set in order to return optimal results. You can find in the full flow including the data download, processing and synthesizing. this notebook Measuring synthetic data fidelity and utility Now that we were able to synthesize our data, it’s time to check whether the new data is able to reproduce properly the behavior observed in the original stock data data. Visual comparison One of my favorites ways to compare real with synthetic data is through visualization. Of course that in terms of automation is not the ideal to validate the quality of the new synthetic data, but it gives us already a pretty good idea. To ensure a 2D visualization of the results, it was applied both a and a with 2 components. TSNE PCA The results are pretty promising, as we see an almost perfect overlap between the synthetic and the real data points. Synthetic data utility — Train synthetic test real There are a plenty of different metrics that can be used to measure the utility of the data such as the . SRA and the TSTR In this example we’ve decided to go for the to validate the utility of our synthetic data. To be used as a regressor model, I’ve decided for a simple architecture with a single layer GRU with 12 units. TSTR method opt = Adam(name= )
    loss = MeanAbsoluteError(name= )
    model = Sequential()
    model.add(GRU(units=units,
                  name= ))
    model.add(Dense(units= ,
                    activation= ,
                    name= ))
    model.compile(optimizer=opt, loss=loss) model : def RNN_regression (units) 'AdamOpt' 'MAE' f'RNN_1' 6 'sigmoid' 'OUT' return Prior to train the model, we had to prepare the inputs: 75% of the data was used as train data (for both synthetic and real datasets) 25% of the real data was used as the test set. For both the model trained on synthetic and the one trained on the real data, I’ve used as the validation set loss as the early stopping criteria. Below are the summarized results obtained for the test set. Conclusion The results obtained with this experiment are very promising and exciting in what concerns the generation of synthetic sequential data. Nevertheless, there are a few caveats that are needed to be pointed: the data under study can be considered to be quite simple, as it was relatively small, with a daily time frequency, there were no surprises in terms of missing values and the dimensionality (columns wise) was relatively low. Not to mention, the very intensive training time. But without a doubt, Generative Adversarial Networks are impressive frameworks, that can be leveraged for much more than synthetic data generation. For those of you, that are looking to keep exploring synthetic data generation have a look into our . We’ll be updating it with new generative algorithms as well as new data, and we invite you to collaborate! GitHub repository Fabiana Clemente is CDO at YData . Improved and synthetic data for AI. YData provides the first dataset experimentation platform for Data Scientists. Also published here

A GAN approach To Synthetic Time-Series Data

This is the first episode of a podcast series on Machine Learning and Data privacy. Machine Learning is the key to the new revolution in many industries. Nevertheless, ML does not exist without data and a lot of it, which in many cases results in the use of sensitive information. With new privacy regulations, access to data is today harder and much more difficult but, does that mean that ML and Data Science has its days counted? Will the machines beat privacy? Don’t forget to subscribe to the and to give a star to the . mlops.community slack Synthetic data open source repo Useful links: In case you’re curious about GDPR fines - enforcementtracker.com The Netflix Prize - https://www.nytimes.com/2010/03/13/te... Tensorflow privacy - https://github.com/tensorflow/privacy YData offers a dataset experimentation platform with synthetic data generation that makes the process of building datasets take a fraction of the time and cost they used to.

Podcast - When Machine Learning Meets Privacy

This is a sentence that is getting too common, but it’s still true and reflects the market's trend, Some of the biggest players in the market already have the strongest hold on that currency. A synthetic data generation dedicated repository. Data is the new oil . When it comes to Machine Learning, definitely data is a pre-requisite, and although the entry barrier to the world of algorithms is nowadays lower than before, there are still a lot of barriers in what concerns, the data use in real-world problems — sometimes access is restricted, others there’s not enough data to get good results, the variability is not enough for model’s generalization, and the list goes on. For those reasons, is one of the new must-have-skills for data scientists! synthetic data generation What is synthetic data after all? Synthetic data can be defined as any data that was not collected from real-world events, meaning, is generated by a system, with the aim to mimic real data in terms of essential characteristics. There are specific algorithms that are designed and able to that can be used as a training dataset. Synthetic data provides several benefits: privacy, as all the personal information has been removed and the data is not possible to be traced back to being less costly and faster to collect when compared to collecting real-world data. generate realistic synthetic data The algorithms you must know and need! There are several different methods to generate synthetic data, some of them very familiar to data science teams, such as . Nevertheless, when it comes to generating realistic synthetic data, we shall have a look into other familiar algorithms — Deep Generative Networks. The repository I’ll be covering in this blog post is a . SMOTE or ADYSIN compilation of different generative algorithms to generate synthetic data Generating synthetic data with WGAN The Wasserstein GAN is considered to be an extension of the Generative Adversarial network introduced by . WGAN was introduced by Martin Arjovsky in 2017 and promises to improve both the stability when training the model as well as introduces a loss function that is able to correlate with the quality of the generated events. ? But what are the core differences introduced with WGAN? Ian Goodfellow Sounds good right What’s new in WGAN? Instead of using the concept of “Discriminator” to classify or predict the probability of a certain generated event as being real or fake, this GAN introduces the concept of a “Critic” that in a nutshell, scores the realness or fakeness of a given event. This change is mainly due to the fact that while training a generator, theoretically we should seek the minimization of the distance between the distribution of the data observed in the training dataset and the distribution observed in the generated examples. We can summarize the major differences introduced by WGAN as the following: Use a new loss function derived from the Wasserstein distance. After every gradient update on the critic function, clamp the weights to a small fixed range, [−c,c]. This allows to enforce the Lipschitz constraint. Alternative proposed to the Discriminator — the Critic. Use of a linear activation function as the output linear of the Critic network. Different number of updates for the Generator and Critic Networks. The benefits The changes mentioned before and introduced with WGAN brings a series of benefits while training these networks: The training of a WGAN is more stable when compared to, for example, the original proposed GAN. It is less sensitive to model architecture selection (Generator and Critic choice) Also, less sensitive and impacted by the hyperparameters choice — although this is still very important to achieve good results. Finally, we are able to correlate the loss of the critic with the overall quality of the generated events. Implementation with Tensorflow 2.0 Now what we’ve completed the imports let’s go for the networks: the Generator and the Critic. Similarly to the Generator, I’ve decided to go for a simple Network for the Critic. Here I’ve a 4 Dense layers network with also Relu activation. But, I want to emphasize a bit here the last code line. Different from Vanilla GAN where we add we usually add this as the last layer of the network: x = Dense( , activation= )(x)) 1 'sigmoid' It uses the sigmoid function in the output layer of the discriminator, which means that it predicts the likelihood of a given event to be real. When it comes to WGAN, the critic model requires a linear activation, in order to predict the score of the “realness” for a given event. x = Dense( )(x) 1 or x = Dense( , activation= )(x) 1 'linear' As I’ve mentioned, the main contribution of the WGAN model is the use of a new loss function — The Wasserstein loss. In this case, we can implement the Wasserstein loss as a custom function in Keras, which calculates the average score for the real and generated events. The score is maximizing the real events and minimizing the generated ones. Below the implementation of Wasserstein loss: K.mean(y_true * y_pred) : def wasserstein_loss (self, y_true, y_pred) return Another important change is the introduction of the weight clipping for the Critic Network. In this case, I’ve decided to defined to extend the Keras constraint class, with the below method: self.clip_value = clip_value backend.clip(weights, -self.clip_value, self.clip_value) { : self.clip_value} #https://keras.io/api/layers/constraints/ : class ClipConstraint (constraints.Constraint) # set clip value when initialized : def __init__ (self, clip_value) : def __call__ (self, weights) return # get the config : def get_config (self) return 'clip_value' Now that we’ve covered the major changes, you can find the full implementation of WGAN in this open GitHub repository, supported by YData. The Challenges with WGAN Although WGAN brings a lot of benefits to the area of data generation, it still has some challenges that need to be solved: Still suffers from unstable training, although being more stable when compared to other architectures Slow convergence after weight clipping — when the clipping window is too large Vanishing gradients — when clipping window is too small Since its publication, and based on the fact that WGAN major issues are related to the chosen method for weights clipping, some have been the suggested improvements, being one of the most promising and used the gradient penalties — . WGAN-GP article Tabular data generation Now that we’ve covered the most theoretical bits about WGAN as well as its implementation, let’s jump into its use to generate synthetic tabular data. For the purpose of this exercise, I’ll use the implementation of WGAN from the repository that I’ve mentioned previously in this blog post. The dataset that I’ll be using for this purpose is one pretty familiar among the Data Science community — the . Credit Fraud dataset For the purpose of demonstration, I’ve decided to go for a smaller sample of this dataset. In this case, I’ve decided to only synthesize fraudulent events. After a few , we are ready to feed our data into WGAN. preprocessing steps on the data Update the Critic more times than the Generator In other GAN architectures, such as , both the generator and the discriminator model must be updated in an equal amount of time. But this is not entirely true for the WGAN. In this case, the critic model must be updated more times than the generator model. DCGAN That’s why we have an input parameter, that I’ve called the — this parameter controls the number of times the critic gets update from every batch of the generator. In this case, I’ve set it to 3 times. But you can set for others and check the impacts in the end results. n_critic When compared to the training process of other , while using the same dataset, it is possible to perceive that indeed WGAN training was less prone to instability and producing results that are closer to the real dataset distribution, although not perfect. GAN architectures I’ve also decided to reduce the dimensionality of the dataset, by leveraging both PCA and TSNE algorithms with the choice of 2 components, in order to ease the visualization of the data. Below you can find the plots, where I compare the results of both PCA and TSNE for the WGAN generated data and the original one. It is clear, that WGAN failed to fit some of the behavior captured in the original data, nevertheless, the results are quite promising. Conclusions The results shown in this blog are still very simple, in comparison with what can be done and achieved with generative algorithms to generate synthetic data with real-value that can be used as training data for Machine Learning tasks. For those who want to know more about generating synthetic data and want to have a try, have a look into this . We’ll be updating it with new generative algorithms as well as new dataset examples, and we invite you to collaborate! GitHub repository

How to Generate Synthetic Data?

A Data-centric Approach to Class-specific Bias in Image Data Augmentation: Conclusion and Limitation

DIY Fake News Detector: Unmask misinformation with Recurrent Neural Networks

Score Big With Football Analytics: Design Your Country's Perfect National Team with Machine Learning

An Overview of the Data-Loader Landscape: Conclusion, Acknowledgments, and References

ডেটা-লোডার ল্যান্ডস্কেপের একটি ওভারভিউ: উপসংহার, স্বীকৃতি এবং রেফারেন্স

Ein Überblick über die Data-Loader-Landschaft: Fazit, Danksagungen und Referenzen

Una descripción general del panorama del cargador de datos: conclusión, agradecimientos y referencias

Un aperçu du paysage des chargeurs de données : conclusion, remerciements et références

डेटा-लोडर परिदृश्य का अवलोकन: निष्कर्ष, आभार और संदर्भ

データローダーの概要: 結論、謝辞、参考文献

Data-Loader 환경 개요: 결론, 승인 및 참고 자료

Uma visão geral do cenário do Data-Loader: conclusão, agradecimentos e referências

Обзор ландшафта загрузчиков данных: заключение, благодарности и ссылки

Veri Yükleyici Ortamına Genel Bakış: Sonuç, Teşekkür ve Referanslar

Tổng quan về bối cảnh trình tải dữ liệu: Kết luận, lời cảm ơn và tài liệu tham khảo

数据加载器概况：结论、致谢和参考文献

Authors:
(1) Troisemaine Colin, Department of Computer Science, IMT Atlantique, Brest, France., and Orange Labs, Lannion, France;
(2) Reiffers-Masson Alexandre, Department of Computer Science, IMT Atlantique, Brest, France.;
(3) Gosselin Stephane, Orange Labs, Lannion, France;
(4) Lemaire Vincent, Orange Labs, Lannion, France;
(5) Vaton Sandrine, Department of Computer Science, IMT Atlantique, Brest, France. Table of Links Abstract and Intro Related work Approaches Hyperparameter optimization Estimating the number of novel classes Full training procedure Experiments Conclusion Declarations References Appendix A: Additional result metrics Appendix B: Hyperparameters Appendix C: Cluster Validity Indices numerical results Appendix D: NCD k-means centroids convergence study 8 Conclusion In this article, we have shown that in the NCD setting, unsupervised clustering algorithms can benefit from knowledge of the known classes and reliably improve their performance by implementing simple modifications. We have also introduced a novel NCD algorithm called PBN, which is characterized by its simplicity and low number of hyperparameters, which proved to be a decisive advantage under realistic conditions. In addition, we have proposed an adaptation of the k-fold cross-validation process to tune the hyperparameters of NCD methods without depending on the labels of the novel classes. Finally, we have demonstrated that the number of novel classes can be accurately estimated within the latent space of PBN. These two previous contributions have shown that the NCD problem can be solved in realistic situations where no prior knowledge of the novel classes is available during training. Declarations Funding Colin Troisemaine, Alexandre Reiffers-Masson, Stephane Gosselin, Vincent Lemaire and Sandrine Vaton received funding from Orange SA. Competing Interests Colin Troisemaine, St´ephane Gosselin and Vincent Lemaire received research support from Orange SA. Alexandre Reiffers-Masson and Sandrine Vaton received research support from IMT Atlantique. Ethics approval Not applicable. Consent to participate All authors have read and approved the final manuscript. Consent for publication Not applicable. Availability of data and materials All data used is this study are available publicly online. The datasets were extracted directly in the repositories available with the links in the corresponding section. Code availability The code for experiments is available at the following url: https://github.com/ PracticalNCD/ECMLPKDD2024. Authors’ contributions Colin Troisemaine, Alexandre Reiffers-Masson, St´ephane Gosselin, Vincent Lemaire and Sandrine Vaton contributed to the manuscript equally. References [1] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: A largescale hierarchical image database. In: CVPR, pp. 248–255 (2009) [2] Troisemaine, C., Lemaire, V., Gosselin, S., Reiffers-Masson, A., Flocon-Cholet, J., Vaton, S.: Novel class discovery: an introduction and key concepts. ArXiv (2023) [3] Hsu, Y.-C., Lv, Z., Kira, Z.: Learning to cluster in order to transfer across domains and tasks. In: ICLR (2018) [4] Han, K., Rebuffi, S.-A., Ehrhardt, S., Vedaldi, A., Zisserman, A.: Autonovel: Automatically discovering and learning novel visual categories. PAMI (2021) [5] Zhong, Z., Zhu, L., Luo, Z., Li, S., Yang, Y., Sebe, N.: Openmix: Reviving known knowledge for discovering novel visual categories in an open world. In: CVPR, pp. 9462–9470 (2021) [6] Zhong, Z., Fini, E., Roy, S., Luo, Z., Ricci, E., Sebe, N.: Neighborhood contrastive learning for novel class discovery. In: CVPR (2021) [7] Sun, Y., Shi, Z., Liang, Y., Li, Y.: When and how does known class help discover unknown ones? provable understanding through spectral analysis. In: ICML, vol. 202, pp. 33014–33043 (2023) [8] Li, Z., Otholt, J., Dai, B., Hu, D., Meinel, C., Yang, H.: A closer look at novel class discovery from the labeled set. In: NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications (2022) [9] Troisemaine, C., Flocon-Cholet, J., Gosselin, S., Vaton, S., Reiffers-Masson, A., Lemaire, V.: A method for discovering novel classes in tabular data. In: ICKG, pp. 265–274 (2022) [10] Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV, pp. 1–21 (2021) [11] Vaze, S., Han, K., Vedaldi, A., Zisserman, A.: Generalized category discovery. In: CVPR, pp. 7492–7501 (2022) [12] Fei, Y., Zhao, Z., Yang, S., Zhao, B.: Xcon: Learning with experts for fine-grained category discovery. In: British Machine Vision Conference (BMVC) (2022) [13] Zhang, L., Qi, L., Yang, X., Qiao, H., Yang, M.-H., Liu, Z.: Automatically Discovering Novel Visual Categories with Self-supervised Prototype Learning (2022) [14] Chen, Y., Zhu, X., Li, W., Gong, S.: Semi-supervised learning under class distribution mismatch. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 3569–3576 (2020) [15] Guo, L.-Z., Zhang, Z.-Y., Jiang, Y., Li, Y.-F., Zhou, Z.-H.: Safe deep semisupervised learning for unseen-class unlabeled data. In: ICML, pp. 3897–3906 (2020) [16] Han, K., Vedaldi, A., Zisserman, A.: Learning to discover novel visual categories via deep transfer clustering. In: ICCV (2019) [17] Hsu, Y.-C., Lv, Z., Schlosser, J., Odom, P., Kira, Z.: Multi-class classification without multi-class labels. In: ICLR (2019) [18] Zheng, J., Li, W., Hong, J., Petersson, L., Barnes, N.: Towards open-set object detection and discovery. In: CVPR, pp. 3961–3970 (2022) [19] Yang, M., Zhu, Y., Yu, J., Wu, A., Deng, C.: Divide and conquer: Compositional experts for generalized novel class discovery. In: CVPR, pp. 14268–14277 (2022) [20] Cao, K., Brbic, M., Leskovec, J.: Open-world semi-supervised learning. In: ICLR (2022) [21] Sun, Y., Li, Y.: Opencon: Open-world contrastive learning. In: TMLR (2023) [22] Chi, H., Liu, F., Yang, W., Lan, L., Liu, T., Han, B., Niu, G., Zhou, M., Sugiyama, M.: Meta discovery: Learning to discover novel classes given very limited data. In: ICLR (2022) [23] Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: ICLR (2018) [24] Arthur, D., Vassilvitskii, S.: K-means++ the advantages of careful seeding. In: ACM-SIAM SODA, pp. 1027–1035 (2007) [25] Luxburg, U.: A tutorial on spectral clustering. Stat. Comput. 17(4), 395–416 (2007) [26] Ng, A., Jordan, M., Weiss, Y.: On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems 14 (2001) [27] Khan, A.A., Mohanty, S.K.: A fast spectral clustering technique using mst based proximity graph for diversified datasets. Information Sciences 609, 1113–1131 (2022) [28] Stuetzle, W.: Estimating the cluster tree of a density by analyzing the minimal spanning tree of a sample. Journal of classification 20(1), 25–47 (2003) [29] Ester, M., Kriegel, H.-P., Sander, J., Xu, X., et al.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Kdd, vol. 96, pp. 226–231 (1996) [30] Le, L., Patterson, A., White, M.: Supervised autoencoders: Improving generalization performance with unsupervised regularizers. Advances in neural information processing systems 31 (2018) [31] Zhao, B., Han, K.: Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. In: Advances in Neural Information Processing Systems (2021) [32] Xie, J., Girshick, R., Farhadi, A.: Unsupervised deep embedding for clustering analysis. In: ICML, vol. 48, pp. 478–487 (2016) [33] Arbelaitz, O., Gurrutxaga, I., Muguerza, J., P´erez, J.M., Perona, I.: An extensive comparative study of cluster validity indices. Pattern recognition 46(1), 243–256 (2013) [34] Yang, M., Wang, L., Deng, C., Zhang, H.: Bootstrap your own prior: Towards distribution-agnostic novel class discovery. In: CVPR, pp. 3459–3468 (2023) [35] Anguita, D., Ghio, A., Oneto, L., Parra, X., Reyes-Ortiz, J.L.: A public domain dataset for human activity recognition using smartphones. In: ESANN (2013) [36] Frey, P.W., Slate, D.J.: Letter recognition using holland-style adaptive classifiers. Machine Learning 6, 161–182 (2005) [37] Dua, D., Graff, C.: UCI Machine Learning Repository (2017) [38] Kuhn, H.W., Yaw, B.: The hungarian method for the assignment problem. Naval Res. Logist. Quart, 83–97 (1955) [39] Von Luxburg, U., Williamson, R.C., Guyon, I.: Clustering: Science or art? In: ICML Workshop on Unsupervised and Transfer Learning, pp. 65–79 (2012) [40] Satopaa, V., Albrecht, J., Irwin, D., Raghavan, B.: Finding a” kneedle” in a haystack: Detecting knee points in system behavior. In: ICDCS Workshops, pp. 166–171 (2011). IEEE [41] Arvai, K.: kneed. Zenodo (2023). https://doi.org/10.5281/zenodo.7873825 This paper is available on arxiv under CC 4.0 license. Authors: (1) Troisemaine Colin, Department of Computer Science, IMT Atlantique, Brest, France., and Orange Labs, Lannion, France; (2) Reiffers-Masson Alexandre, Department of Computer Science, IMT Atlantique, Brest, France.; (3) Gosselin Stephane, Orange Labs, Lannion, France; (4) Lemaire Vincent, Orange Labs, Lannion, France; (5) Vaton Sandrine, Department of Computer Science, IMT Atlantique, Brest, France. Authors: Authors: (1) Troisemaine Colin, Department of Computer Science, IMT Atlantique, Brest, France., and Orange Labs, Lannion, France; (2) Reiffers-Masson Alexandre, Department of Computer Science, IMT Atlantique, Brest, France.; (3) Gosselin Stephane, Orange Labs, Lannion, France; (4) Lemaire Vincent, Orange Labs, Lannion, France; (5) Vaton Sandrine, Department of Computer Science, IMT Atlantique, Brest, France. Table of Links Abstract and Intro Abstract and Intro Related work Related work Approaches Approaches Hyperparameter optimization Hyperparameter optimization Estimating the number of novel classes Estimating the number of novel classes Full training procedure Full training procedure Experiments Experiments Conclusion Conclusion Declarations Declarations References References Appendix A: Additional result metrics Appendix A: Additional result metrics Appendix B: Hyperparameters Appendix B: Hyperparameters Appendix C: Cluster Validity Indices numerical results Appendix C: Cluster Validity Indices numerical results Appendix D: NCD k-means centroids convergence study Appendix D: NCD k-means centroids convergence study 8 Conclusion In this article, we have shown that in the NCD setting, unsupervised clustering algorithms can benefit from knowledge of the known classes and reliably improve their performance by implementing simple modifications. We have also introduced a novel NCD algorithm called PBN, which is characterized by its simplicity and low number of hyperparameters, which proved to be a decisive advantage under realistic conditions. In addition, we have proposed an adaptation of the k-fold cross-validation process to tune the hyperparameters of NCD methods without depending on the labels of the novel classes. Finally, we have demonstrated that the number of novel classes can be accurately estimated within the latent space of PBN. These two previous contributions have shown that the NCD problem can be solved in realistic situations where no prior knowledge of the novel classes is available during training. Declarations Funding Colin Troisemaine, Alexandre Reiffers-Masson, Stephane Gosselin, Vincent Lemaire and Sandrine Vaton received funding from Orange SA. Competing Interests Colin Troisemaine, St´ephane Gosselin and Vincent Lemaire received research support from Orange SA. Alexandre Reiffers-Masson and Sandrine Vaton received research support from IMT Atlantique. Ethics approval Not applicable. Consent to participate All authors have read and approved the final manuscript. Consent for publication Not applicable. Availability of data and materials All data used is this study are available publicly online. The datasets were extracted directly in the repositories available with the links in the corresponding section. Code availability The code for experiments is available at the following url: https://github.com/ PracticalNCD/ECMLPKDD2024. Authors’ contributions Colin Troisemaine, Alexandre Reiffers-Masson, St´ephane Gosselin, Vincent Lemaire and Sandrine Vaton contributed to the manuscript equally. References [1] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: A largescale hierarchical image database. In: CVPR, pp. 248–255 (2009) [2] Troisemaine, C., Lemaire, V., Gosselin, S., Reiffers-Masson, A., Flocon-Cholet, J., Vaton, S.: Novel class discovery: an introduction and key concepts. ArXiv (2023) [3] Hsu, Y.-C., Lv, Z., Kira, Z.: Learning to cluster in order to transfer across domains and tasks. In: ICLR (2018) [4] Han, K., Rebuffi, S.-A., Ehrhardt, S., Vedaldi, A., Zisserman, A.: Autonovel: Automatically discovering and learning novel visual categories. PAMI (2021) [5] Zhong, Z., Zhu, L., Luo, Z., Li, S., Yang, Y., Sebe, N.: Openmix: Reviving known knowledge for discovering novel visual categories in an open world. In: CVPR, pp. 9462–9470 (2021) [6] Zhong, Z., Fini, E., Roy, S., Luo, Z., Ricci, E., Sebe, N.: Neighborhood contrastive learning for novel class discovery. In: CVPR (2021) [7] Sun, Y., Shi, Z., Liang, Y., Li, Y.: When and how does known class help discover unknown ones? provable understanding through spectral analysis. In: ICML, vol. 202, pp. 33014–33043 (2023) [8] Li, Z., Otholt, J., Dai, B., Hu, D., Meinel, C., Yang, H.: A closer look at novel class discovery from the labeled set. In: NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications (2022) [9] Troisemaine, C., Flocon-Cholet, J., Gosselin, S., Vaton, S., Reiffers-Masson, A., Lemaire, V.: A method for discovering novel classes in tabular data. In: ICKG, pp. 265–274 (2022) [10] Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV, pp. 1–21 (2021) [11] Vaze, S., Han, K., Vedaldi, A., Zisserman, A.: Generalized category discovery. In: CVPR, pp. 7492–7501 (2022) [12] Fei, Y., Zhao, Z., Yang, S., Zhao, B.: Xcon: Learning with experts for fine-grained category discovery. In: British Machine Vision Conference (BMVC) (2022) [13] Zhang, L., Qi, L., Yang, X., Qiao, H., Yang, M.-H., Liu, Z.: Automatically Discovering Novel Visual Categories with Self-supervised Prototype Learning (2022) [14] Chen, Y., Zhu, X., Li, W., Gong, S.: Semi-supervised learning under class distribution mismatch. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 3569–3576 (2020) [15] Guo, L.-Z., Zhang, Z.-Y., Jiang, Y., Li, Y.-F., Zhou, Z.-H.: Safe deep semisupervised learning for unseen-class unlabeled data. In: ICML, pp. 3897–3906 (2020) [16] Han, K., Vedaldi, A., Zisserman, A.: Learning to discover novel visual categories via deep transfer clustering. In: ICCV (2019) [17] Hsu, Y.-C., Lv, Z., Schlosser, J., Odom, P., Kira, Z.: Multi-class classification without multi-class labels. In: ICLR (2019) [18] Zheng, J., Li, W., Hong, J., Petersson, L., Barnes, N.: Towards open-set object detection and discovery. In: CVPR, pp. 3961–3970 (2022) [19] Yang, M., Zhu, Y., Yu, J., Wu, A., Deng, C.: Divide and conquer: Compositional experts for generalized novel class discovery. In: CVPR, pp. 14268–14277 (2022) [20] Cao, K., Brbic, M., Leskovec, J.: Open-world semi-supervised learning. In: ICLR (2022) [21] Sun, Y., Li, Y.: Opencon: Open-world contrastive learning. In: TMLR (2023) [22] Chi, H., Liu, F., Yang, W., Lan, L., Liu, T., Han, B., Niu, G., Zhou, M., Sugiyama, M.: Meta discovery: Learning to discover novel classes given very limited data. In: ICLR (2022) [23] Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: ICLR (2018) [24] Arthur, D., Vassilvitskii, S.: K-means++ the advantages of careful seeding. In: ACM-SIAM SODA, pp. 1027–1035 (2007) [25] Luxburg, U.: A tutorial on spectral clustering. Stat. Comput. 17(4), 395–416 (2007) [26] Ng, A., Jordan, M., Weiss, Y.: On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems 14 (2001) [27] Khan, A.A., Mohanty, S.K.: A fast spectral clustering technique using mst based proximity graph for diversified datasets. Information Sciences 609, 1113–1131 (2022) [28] Stuetzle, W.: Estimating the cluster tree of a density by analyzing the minimal spanning tree of a sample. Journal of classification 20(1), 25–47 (2003) [29] Ester, M., Kriegel, H.-P., Sander, J., Xu, X., et al.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Kdd, vol. 96, pp. 226–231 (1996) [30] Le, L., Patterson, A., White, M.: Supervised autoencoders: Improving generalization performance with unsupervised regularizers. Advances in neural information processing systems 31 (2018) [31] Zhao, B., Han, K.: Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. In: Advances in Neural Information Processing Systems (2021) [32] Xie, J., Girshick, R., Farhadi, A.: Unsupervised deep embedding for clustering analysis. In: ICML, vol. 48, pp. 478–487 (2016) [33] Arbelaitz, O., Gurrutxaga, I., Muguerza, J., P´erez, J.M., Perona, I.: An extensive comparative study of cluster validity indices. Pattern recognition 46(1), 243–256 (2013) [34] Yang, M., Wang, L., Deng, C., Zhang, H.: Bootstrap your own prior: Towards distribution-agnostic novel class discovery. In: CVPR, pp. 3459–3468 (2023) [35] Anguita, D., Ghio, A., Oneto, L., Parra, X., Reyes-Ortiz, J.L.: A public domain dataset for human activity recognition using smartphones. In: ESANN (2013) [36] Frey, P.W., Slate, D.J.: Letter recognition using holland-style adaptive classifiers. Machine Learning 6, 161–182 (2005) [37] Dua, D., Graff, C.: UCI Machine Learning Repository (2017) [38] Kuhn, H.W., Yaw, B.: The hungarian method for the assignment problem. Naval Res. Logist. Quart, 83–97 (1955) [39] Von Luxburg, U., Williamson, R.C., Guyon, I.: Clustering: Science or art? In: ICML Workshop on Unsupervised and Transfer Learning, pp. 65–79 (2012) [40] Satopaa, V., Albrecht, J., Irwin, D., Raghavan, B.: Finding a” kneedle” in a haystack: Detecting knee points in system behavior. In: ICDCS Workshops, pp. 166–171 (2011). IEEE [41] Arvai, K.: kneed. Zenodo (2023). https://doi.org/10.5281/zenodo.7873825 This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Stories, News & Mentions about YDATA

Ydata’s Stories on HackerNoon

Ydata's News & Mentions