texts: Python equivalent of Wolfram Language ParametricPlot3D?

texts:
The Wolfram Language as a ParametricPlot3D that can be used  to interactively examine parametric functions in 3D. Which Python package and function(s) best replicates this?

I have several series of data that I would like to represent as categorical curves in an interactive 3D plot in a Python project.

As a minimal example on a set of 10 series with 20 observations in each, in Wolfram Language I can

SeedRandom[987]
obs = RandomVariate[StudentTDistribution[3, 1, 5], {10, 20}];
foos = PDF@*SmoothKernelDistribution /@ obs;

ParametricPlot3D[
 MapIndexed[{First@#2, u, v #1[u]} &]@foos, {u, -5, 10}, {v, 0, 1},
 BoxRatios -> {5, 1, 1},
 PlotRange -> Full,
 ColorFunction -> ColorData["SunsetColors"],
 Background -> Lighter@Purple,
 AxesStyle -> White,
 TicksStyle -> White,
 Boxed -> False,
 AxesEdge -> {{0, 0}, {1, 0}, {1, 1}},
 ViewPoint -> {3, -2, 1.5},
 ImageSize -> Large,
 Mesh -> None,
 PlotLegends -> Automatic,
 PlotPoints -> {80, 15},
 Ticks -> {{#, IntegerName@#} & /@ Range@10, Automatic, Automatic}]

How is this done with Python?


texts: Unsupervised document similarity state of the art

texts:
I have a set of N documents with lengths ranging from 0 to more than 20000 characters. I want to calculate a similarity score between 0 and 1 between all pairs of documents where a higher number indicates higher similarity. Assume below that deploying a supervised model is infeasible due to resource constraints that are not necessarily data science related (gathering labels is expensive, infrastructure cannot be approved for supervised models for whatever reason etc).
Approaches I have considered:

tf-idf
Smooth Inverse Frequency (SIF) embeddings and its developments (uSIF, p-SIF). https://openreview.net/pdf?id=SyK00v5xx https://www.aclweb.org/anthology/W18-3012/ https://arxiv.org/abs/2005.09069
BERT or bert-like embeddings, e.g., https://arxiv.org/abs/2010.06467
Hierarchical Optimal Transport for Document Representation (HOTT): https://papers.nips.cc/paper/2019/hash/8b5040a8a5baf3e0e67386c2e3a9b903-Abstract.html

Question:
Is there an unsupervised technique that has been shown in a peer-reviewed setting to achieve higher accuracy (or F1 or similar) on long texts (more than, say, 10000 characters) than HOTT?
Background:
The HOTT paper benchmarks various approaches with a k-NN classifier and shows that HOTT performs best, but not dramatically better than tf-idf (HOTT has 0.52 vs tf-idf's 0.66 normalized error). Note that while the HOTT algorithm is unsupervised the datasets in the paper are labeled, otherwise a benchmark would not be possible.
The SIF papers mostly deal with the STS datasets which are not long texts. p-SIF has a benchmark on the Reuters dataset, but uses a SVM supervised approach. Interestingly, the HOTT paper finds that SIF does not perform well with the k-NN approach with 0.79 normalized error.
In many cases BERT requires pre-training and if it does not, its max or average pooled performance appears to be worse than glove embeddings (https://arxiv.org/abs/2010.06467 page 114).
I have also not been able to find unsupervised benchmarks for Doc2Vec, Universal Sentence Encoder (USE).
There is additionally the question of how to calculate the similarity once the embedding is obtained (e.g., https://www.aclweb.org/anthology/N19-1100.pdf) but that is out of scope for this question unless it affects comparison between unsupervised benchmarks (e.g., the k-NN approach can use various distance metrics which may affect accuracy).
If the benchmarks in HOTT are representative and no other methods exist that perform substantially better it is tempting to make the conclusion that tf-idf is still a strong approach since it is so simple to implement and understand (it is certainly simpler than HOTT). If that is the case I think it is a remarkable conclusion given the deep learning developments in the last 5-10 years.
Related posts that do not specifically address this question:
What is considered short and long text in NLP (document similarity)
Alternatives to TF-IDF and Cosine Similarity when comparing documents of differing formats
How to measure the similarity between two text documents?
Cluster documents based on topic similarity
Document similarity: Vector embedding versus BoW performance?
Word2Vec - document similarity
Weighted sum of word vectors for document similarity
Document similarity
Cosine similarity between query and document confusion
Evaluate document similarity / content-based recommender system
Use embeddings to find similarity between documents


texts: Unsupervised Clustering high dimentional data not having estimation for K

texts:
I have a dataset (all numerical) of 50K records containing 500 features. we are trying to find fingerprints. Meaning that we would like to cluster the data and report one of the nodes in each cluster as representative of that cluster (meaning that each node in that cluster is mostly similar to that representative compared to any other node in other clusters). So we won't have any noise (meaning that all nodes should be represented). I have done K-means, Kmedoids, hierarchical, dbscan, hdbscan, etc. Each having their own issues.

Kmeans doesn't report one of the nodes as center for the cluster but the center point which might not be one of our nodes. So we switched to Kmedoids. which the results are too dependent of the initial seed. Also not knowing K is another problem.
Then we used elbow method to find K. But since not sure what is the upper limit for K, the plot is not very Elbow-y but more like a gradual decrease. Then we tried Sillouette0score method and I got WAY too big number for clusters (something over 200 which doesn't seem right to the field expert). Same problem with the number for K generated by Affinity Propagation (too big of K).
Since we have tens of thousands of data, Meanshift doesn't relly work properly. Also HDBSCAN reports 264 labels which again seems unreasonable.

I would like to try some dimensionality reduction methods for this data. But not sure what would work well. tSNE is too focused on visualization which doesn't seem best fit for our use. Any advice would be VERY appreciated.


texts:
I want to compute the "distance" between users in order to return the top n similar users, for any given user.
For each user a have a bunch of features.

This is close to a recommendation system, however I don't have ratings and I can't assign each user to different products, hence I'll need to compare "how similar" Users are.

Do you have any suggestion for the algorithms/approaches for this kind of problem?
I find a lot of resources based on recommendation system (having rating and items), but that's different from what I want to achieve


texts: Which is the best method for Neural Network Layers in Keras

texts:
In keras we can create neural network layers in many ways.
1. Sequential API.
for example
model=sequential()

2. Functional
for example
x1=Input(shape=(2,)
x2=Dense(2)(x1)

3. Subclassing
for example
class Mymodel(keras.layers.Layers):
    def __init__(self,output):

My Question is:
1. Whether which one to use and their pros and cons.


texts: Why is there are no Volcano-plot or MA-plot in bioinformatics widget?


texts: How does sigmoid saturate with large weights?

texts:
In cs231n course , it is mentioned that

  If the initial weights are too large then most neurons would become
  saturated and the network will barely learn.

How do the neurons get saturated? Large weights may lead to a z (output of saturation) which is not cery close 0 or 1 and so doesn't let z*(1-z) to saturate


texts: Categorical features preprocessing for clustering

texts: Can anyone tell suggest the best practice for clustering data with mixtured features (both with categorical and continuous). I am struggling with a problem; I realized that for all metrics algorithms it is necessary to scale continuous data before clustering, so I used sklearn.preprocessing.StandardScaler(). With my categorical features, I used onehotencoder transformation, but it's not clear if it's necessary also scale these onehotencoded vector's components regarding to the whole data set or just leave them as they are?


texts: Difference between OLS(statsmodel) and Scikit Linear Regression

texts:
I have a question about two different methods from different libraries which seems doing same job. I am trying to make linear regression model.

Here is the code which I using statsmodel library with OLS :

X_train, X_test, y_train, y_test = cross_validation.train_test_split(x, y, test_size=0.3, random_state=1)

x_train = sm.add_constant(X_train)
model = sm.OLS(y_train, x_train)
results = model.fit()

print "GFT + Wiki / GT  R-squared", results.rsquared

This print out GFT + Wiki / GT  R-squared 0.981434611923

and the second one is scikit learn library Linear model method:

model = LinearRegression()
model.fit(X_train, y_train)

predictions = model.predict(X_test)

print 'GFT + Wiki / GT R-squared: %.4f' % model.score(X_test, y_test)

This print out GFT + Wiki / GT R-squared: 0.8543

So my question is the both method prints our R^2 result but one is print out 0.98 and the other one is 0.85.

From my understanding, OLS works with training dataset. So my questions,

Is there a way that work with test data set with OLS ?
Is the traning data set score gives us any meaning(In OLS we didn't use test data set)? From my past knowledge we have to work with test data.
What is the difference between OLS and scikit linear regression. Which one we use for calculating the score of the model ?

Thanks for any help.


texts: Sliding window approach using SVR & LightGBM

texts:
I'm working on a multivariate time series forecast using a couple of ML algorithms (Neural Networks, Support Vector Machines & Gradient boosting algorithms). I need to measure the performance of each model.
I've implemented the 1st model using Tensorflow 2.0. Training & testing data was created using tf.Dataset API.
The data format is (window_data, forecast), where window_data represents a set of 24 timesteps and forecast represents  the next timestep.

Now I need to train 2nd & 3rd model using SVR (LinearSVR to be more precise) and LightGBM.
Is it possible to feed the model with a windowed dataset like in my 1st model?


texts: When to normalize or regularize features in Data Science

texts:
How can I know when I need to use normalization and when regularization of features.
I know that when I have big difference between min and max values in some feature / column that i need to scale data, also if I use SVM or KNN or Clustering i need to scale the data.

But how can I know when to normalize and when to regularize the features?
How can I know that if I have for example 200 features and 10 000 rows?
Is there any way that I can plot my data and see what can I use?


texts: OCR / Text Recognition and Recovery Problem

texts:
I am working on a research project that deals with American military casualties during WWII. Specifically, I am attempting to construct a count of casualties for each service at the county level. There are two sources of data here, each presenting their own challenges.

1. Army and Air Force data. The National Archives hosts lists of Army and Air Force servicemen killed in action by state and county. There are .gif images of the report available online. Here is a sample for several counties in Texas.

I DO NOT need to recover the names or any other information. I simply need to count the number of names (each on its own line, and listed in groups of five) under each County. There are hundreds of these images (50 states - 30-100 for each state).

I have been unable to find an OCR program that can tackle this problem adequately. How would you suggest I approach this challenge? (I have some programming expertise in Python and Java, but would prefer to use any off-the-shelf solutions that may exist).

2. Navy and Marine Core data. This data is organized differently. Each state has long lists of casualties with the address of their next of kin. Here is a sample for Texas again. For these images, I need to BOTH count the number of dead and recover their hometown, which is typically the last word in each entry. I can then match these hometowns to counties and merge with database 1.

Again, the usual OCR programs have proved inadequate. Any help on this (admittedly more difficult) problem would be very much appreciated.

Thank you in advance experts!


texts: How to use SimpleImputer Class to replace missing values with mean values using Python?

texts:
This is my code

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

#Importing Dataset
dataset = pd.read_csv('C:/Users/Rupali Singh/Desktop/ML A-Z/Machine Learning A-Z Template Folder/Part 1 - Data Preprocessing/Data.csv')
print(dataset)
X = dataset.iloc[:, :-1].values
Y = dataset.iloc[:, 3].values
#Missing Data

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(missing_values= np.nan, strategy='mean')
X.fit[:, 1:3] = imputer.fit_transform(X[:, 1:3])
print(X)

My data set:

Country   Age   Salary Purchased
0   France  44.0  72000.0        No
1    Spain  27.0  48000.0       Yes
2  Germany  30.0  54000.0        No
3    Spain  38.0  61000.0        No
4  Germany  40.0      NaN       Yes
5   France  35.0  58000.0       Yes
6    Spain   NaN  52000.0        No
7   France  48.0  79000.0       Yes
8  Germany  50.0  83000.0        No
9   France  37.0  67000.0       Yes

Error Message:

File "C:/Users/Rupali Singh/PycharmProjects/Machine_Learning/data_preprocessing_Template.py", line 15, in <module>
    X.fit[:, 1:3] = imputer.fit_transform(X[:, 1:3])
AttributeError: 'numpy.ndarray' object has no attribute 'fit'


texts: Find the optimal n_estimator by looping the model accuracy indicator in random forest algorithm - python

texts:
i'm trying to find the best n_estimator value on a Random Forest ML model by running this loop:

for i in r:
   RF_model_i = RandomForestClassifier(criterion="gini",   n_estimators=i, oob_score=True)
   RF_model_i.id = [i]  # dynamically add fields to objects
   RF_model_i.fit(X_train, y_train)
   y_predict_i =   RF_model_i.predict(X_test)
   accuracy_i = [accuracy_score(y_test, y_predict_i), i]
   results.append(accuracy_i) # put the result on a list within the for-loop

question #1 ** What i like to understand is if this could be a good way to understand how to decide the n_estimator parameter and
possibly why (i'm not so sure it is)
2.**question #2 If what i'm doing have some sense, could be a good idea extend this loop to all of other main parameters?

What i obtain is this:
a level of accuracy for a number of estimator associated with which i runned the random forest algorithm

Thank you


texts: Using BERT for the first time, what are the two columns of my test_results.tsv?

texts: I followed the steps to feed in both dev, test, train.tsv to the model, trained it, then tried to classify test data, and I only have 1 feature, and the classification is binary, 1 or 0. I assumed my test_results.tsv would have just 1 column, representing the estimated classification. Any advice? Here is a picture of my output.  I thought it would be a column of 1's and 0's showing me its estimated classification.


texts: How to store strings in CSV with new line characters?

texts:
My question is: what are ways I can store strings in a CSV that contain newline characters (i.e. \n), where each data point is in one line?

Sample data

This is a sample of the data I have:

data = [
    ['some text in one line', 1],
    ['text with\nnew line character', 0],
    ['another new\nline character', 1]
]

Target CSV

I want a CSV file, where the first line is "text,category" and every subsequent line is an entry from data.

What I tried

Using the csv package from Python.

import csv
field_names = ['text', 'category']

# Writing
with open('file.csv', 'w+', encoding='utf-8') as file:
    csvwriter = csv.DictWriter(file, field_names)
    csvwriter.writeheader()
    for d in data:
        csvwriter.writerow({'text': d[0], 'category':d[1]})

# Reading
with open('file.csv', 'r', encoding='utf-8') as file:
    csvreader = csv.DictReader(file, field_names)
    data = []
    for line in csvreader:
        data.append([line['text'], line['category']])

I can read and write, but the output file.csv is the following:

  text,category

  some text in one line,1

  "text with

  new line character",0

  "another new

  line character",1

So not one line per data point.


texts: Cross Validation for Different Metrics - Sklearn

texts:
When I am doing cross validation using Python's Sklearn and take the score of different metrics (accuracy, precision, etc.) like this:

result_accuracy = cross_val_score(classifier, X_train, y_train, scoring='accuracy', cv=10)
result_precision = cross_val_score(classifier, X_train, y_train, scoring='precision', cv=10)
result_recall = cross_val_score(classifier, X_train, y_train, scoring='recall', cv=10)
result_f1 = cross_val_score(classifier, X_train, y_train, scoring='f1', cv=10)

Did every execution of cross_val_score() function for different metrics made the same 10 folds of the training data or not? If not, do I need to make the general 10-folds first using KFold, like this:

seed = 7
kf = KFold(n_splits=10, random_state=seed)

result_accuracy = cross_val_score(classifier, X_train, y_train, scoring='accuracy', cv=kf)
result_precision = cross_val_score(classifier, X_train, y_train, scoring='precision', cv=kf)
result_recall = cross_val_score(classifier, X_train, y_train, scoring='recall', cv=kf)
result_f1 = cross_val_score(classifier, X_train, y_train, scoring='f1', cv=kf)


texts:
Can there be Machine learning applications in Architecture field?
Example : New York City 100 years back i.e. Year 1920, every ten years after 1920 and today i.e. Year 2020 images of New York City structures are feed as Input dataset of the Machine learning algorithm to predict the future Modern architecture  of Buildings, Offices, Parks, Museums, Shopping Malls etc look and feel in New York city say 1930, 1940, 1950 etc


texts: Newton-Raphson or EM Algorithm in Python

texts:
Is there any implementation of Newton-Raphson or EM Algorithm? Can I get the source code of it?

I tried googling, but didn't come across any. So asking here.

Thanks!


texts: Physical modelling with neural networks - single output + stack ensemble vs multi-output

texts:
We are trying to replace an existing physical model (8 inputs/7 outputs) with artificial neural networks. The physics behind the existing model is mainly thermodynamics of humid air for air conditioning, with some turbomachinery involved, which yields most likely complex functions between inputs and outputs.

One approach was already done:
single output neural networks (10 NN with same # hidden layers but different parameters like batch size, # epochs, optimizer, etc). Then some sort of stacking ensembled was used: every predictions were  used as new inputs for a single NN to predict a final value.

The accuracy is pretty well, however there are some test datapoints where the absolute errors are high enough to be worried about the predicted value (this could be used for example for air conditioning control strategies, so a bad prediction would result in uncontrolled system).

In order to improve accuracy some colleagues were suggesting to keep it simpler and perform just a multi-output regression with a single neural network.

From the mathematical point of view, I have following questions:

1) would a single output NNs stacking ensembled outperformed the multi-output single NN?
2) is the way of stack ensembling using NN a good approach? I saw some different techniques like arithmetic averaging the inputs

Thanks for your time!
Regards


texts: Replacing the feature variables with encoded variables

texts:
I have several nominal variables which I've encoded using the LabelEncoder() function. Now I want to replace the encoded values in the place of the raw datas of the features in the dataframe. I tried df.replace() but no luck. Below is the code snippet.

nominal_values = [
    'HouseStyle','ExterQual','BsmtQual','BsmtExposure','BsmtFinType1',
    'KitchenQual','GarageType','GarageFinish','GarageQual','GarageCond']

One of the column "KitchenQual" has values as follows:

{kitchenqual :['Ex','Gd','TA','Fa','Po']} representing "Excellent","Good","Typical/Average","Fair","Poor" respectively.

Another Feature "BsmtFinType1" has values as follows:

{BsmtFinType1: ['GLQ','ALQ','BLQ','Rec','LwQ','Unf','NA']} represeting

"Good Living Quarters","Average Liv. Quat.","Below Avg.Liv.Quat,","Avg. rec room", "Low Quality", "Unfinished","No Basement" respectively.

from sklearn.preprocessing import LabelEncoder

lbe = LabelEncoder()
for noms in nominal_values:
    encode = lbe.fit_transform(cat_var[noms])
    cat_var.replace(to_replace=cat_var[noms],
                    value=pd.Series(encode),inplace=True)

I used pd.Series(encode) since the replace function will support only one of the following data structures

Scalar
dict
Series


texts: ML Pipeline on AWS SageMaker: How to create long-running query/preprocessing tasks

texts:
I'm a software engineer transitioning toward machine learning engineering, but need some assistance.
I'm currently using AWS Lambda and Step Functions to run query and preprocessing jobs for my ML pipeline, but am restrained by Lambda's 15m runtime limitation.
We're a strictly AWS shop, so I'm kind of stuck with SageMaker and other AWS tools for the time being. Later on we'll consider experimenting with something like Kubeflow if it looks advantageous enough.
My current process

I have my data scientists write python scripts (in a git repo) for the query and preprocessing steps of a model, and deploy them (via Terraform) as Lambda functions, then use Step Functions to sequence the ML Pipeline steps as a DAG (query -> preprocess -> train -> deploy)
The Query lambda pulls data from our data warehouse (Redshift), and writes the unprocessed dataset to S3
The Preprocessing lambda loads the unprocessed dataset from S3, manipulates it as needed, and writes it as training & validation datasets to a different S3 location
The Train and Deploy tasks use the SageMaker python api to train and deploy the models as SageMaker Endpoints

Do I need to be using Glue and SageMaker Processing jobs? From what I can tell, Glue seems more targeted towards ETLs than for writing to S3, and SageMaker Processing jobs seem a bit more complex to deploy to than Lambda.


texts: Computer Vision: Handling dataset(3D data or scan) with different timesteps

texts:
I'm planning on training a CNN on CT scans for classification. The problem is CT scans are taken slice by slice, and in a typical scan, there could be more than 200 slices. The number of slices in a scan isn't uniform and depend on the scanning machine and age of the person(for whom the scan is taken).

1)How should I make the number of slices uniform for feeding to a deep learning network?

This sorta problem is handled in NLP by padding a chosen vector( or something similar) to sentences which have lengths less than the predefined length and truncating sentences which have lengths greater than the preset length.

2)Can a similar approach be used to make slices(timesteps) uniform or is there a better way?


texts: Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

texts: I am looking for a little clarity on what the policy gradient theorem means. My confusion lies in the fact that the reward $R$ in reinforcement learning is non-differentiable in the policy parameters. As that is the case how does the central objective of policy gradients, finding the gradients of Reward $R$ wrt the parameters of policy function even make sense?


texts: Bias and variance in the model o in the predictions?

texts: This topic confuses me. In the literature or articles, when talking about bias and variance in automatic learning, specifically in cross-validation, do they refer to the high bias (underfitting) and high variance (overfitting) in the model? Or do they refer to the bias and variance of the predictions obtained in the iterations of the cross-validation? How to handle each case?


texts: Analysis of factors that have impact on explanatory variable

texts:
for my master thesis I am analyzing stock outs. I want to find out what factors played key roles that finally led to a stock out or (will) lead to a high stockout probability, for example late production, too long quality checks, late supplier etc.

What method is best to use here? I was thinking about random forest or logistic regression, but maybe there is even something better?

Thanks


texts: Choosing best model produced from different algorithms. Metric produced by cross-validation on the train set or metric produced on the test set?

texts:
I know that choosing between models produced by one algorithm with different hyperparameters the metric for choosing the best one should be the cross-validation on train set.
But what about choosing between models that are produced by different algorithm? By which metric should I compare them, the metric produced by cross-validation on the train set or metric produced on the test set? Also why?


texts: Literature on selecting specific dimensions in a word embedding vector

texts:
I am aware that the different dimensions in the word embedding represents different information and algebraic operations can be performed between two embeddings for example.
Can anyone point me to literature on selecting specific dimensions from a word embedding vector. I don't mean dimensionality reduction, but papers following the theory that all those dimensions are not important for all the tasks and that specific ones will be more important for a specific task. For example: Sentiment Analysis would benefit from dimension 5th to 55th and 250th to 300th instead of using the whole 300 dimensions.
Let me know if this theory isn't true.


texts:
I am learning about Trax for the implementation of GRU and LSTMs.
Their documentation says that a GRU layer in Trax can only accept a number of hidden units equal to the number of elements in the embeddings(Embedding size) of the input words .
Having studied the traditional implementations of RNNs and GRUs, my question is , shouldn't the number of hidden units in the GRU layer be equal to the input dimension? So as to propagate the values of hidden states from say the first word of a sentence upto the last word in recurrence.
If the number of hidden units in GRU are fixed to the embedding size, how does the Recurrence part work?
There seems to be no guide on this in the internet.
Can someone please help me understand how this setting will work ?


texts: Validation accuracy greater than training accuracy in cnn

texts: I've splitted my training set in the ratio 80:20 and have developed cnn model with a dropout of 0.5. I'm getting an accuracy of 98%. But the validation accuracy stays greater than training accuracy. Is there any problem with this?Does it leads to over fitting? If yes, then why am I getting an accuracy of 98%?The graph is given below.The red line is the validation accuracy and blue is training accuracy.


texts: What is the best way (cheapest / fastest option) to train an model on massive dataset (400GB+, 100m rows x 200 columns)?

texts:
I have a 400GB data set that I want to train a model on. What is the cheapest method to train this model? The options I can think of so far are:

AWS instance with massive RAM and train CPU (slow, but instances are cheap).
AWS instance with many GPUs and using Dask + XGBoost to distribute (fast, but expensive instances, and  I don't even think there would be an instance large enough to handle).

I have just assumed XGBoost is the best package since its tabular data, but if another gradient boosted tree package would be better at handling this, that would be acceptable as well.
Any help would be greatly appreciated.


texts:
I got ValueError when predicting test data using a RandomForest model.

My code:

clf = RandomForestClassifier(n_estimators=10, max_depth=6, n_jobs=1, verbose=2)
clf.fit(X_fit, y_fit)

df_test.fillna(df_test.mean())
X_test = df_test.values
y_pred = clf.predict(X_test)

The error:

ValueError: Input contains NaN, infinity or a value too large for dtype('float32').

How do I find the bad values in the test dataset? Also, I do not want to drop these records, can I just replace them with the mean or median?

Thanks.


texts: use genetic algorithm as a feature selection for text classification

texts:
how to apply the genetic algorithm as a feature selection for text classification  in python
I need to use GA to select most relevant feature in text classification


texts: Is feature importance from classification a good way to select features for clustering?

texts: I have a large data set with many features (70). By doing preprocessing (removing features with too many missing values and those that are not correlated with the binary target variable) I have arrived at 15 features. I am now using a decision tree to perform classification with respect to these 15 features and the binary target variable so I can obtain feature importance. Then, I would choose features with high importance to use as an input for my clustering algorithm. Does using feature importance in this context make any sense?


texts: Are Hadoop and Python SciPy used for the same?

texts:
Right now

I've been doing a bit of research because I'm quite new to Big Data world. Among several other tools or frameworks, I've read about Apache Hadoop and Python for data analysis.

Specifically, I've read that:

Hadoop: allows you to perform any task you want, if it fits the Map-Reduce paradigm. It can add concurrence using a cluster, etc.

SciPy: python based ecosystem for (in fact, a lot of things)...

So my question is

If I've data collected from an environment, and want to correlate the data, calculate means, extract conclusions, etc...

Is SciPy as suitable as Hadoop (or more) to do so?

I'm basically trying to choose between them in order to build a little example, so I wonder which one fits better, or may I be misunderstanding the uses of each one.


texts: As I've been introducing myself to the various deep learning frameworks, I've noticed a difference in the default placement of channels for images.  Is there a substantial difference between NCHW vs NHCW layout?  Why would I choose one over the other?


texts: Optimization problem with different type of constraints

texts:
I'm new to optimization problems. I want to find optimum values for my objective function. You can imagine my function as E = f(t1, t2, t3). I want to minimize E and following constraints limit the variables:
1- 0 < t1, t2, t3 < 255
2- t1, t2, and t3 be less as possible
3- being zero for one of this parameters is more important than overall values be low. for example (80, 150, 0) is better than (140, 50, 20) because first set of parameters have zero. Note that nature of E almost reacts as mentioned. However, I want to limit parameters as mentioned if it is possible.
Is there algorithm for such problems? I read optimization in this linear programming example of Matlab but I can't find constraints mentioned in 2 and 3. Any help using Matlab or Python 3.X will be appreciated.


texts: Customized function for Agglomerative Clustering

texts:
I need to use a customized linkage function for my project. For this function, on every step I should calculate the objective containing

similarity (distance) between subclusters to be merged
distances between leaves and roots in terms of edges in the forest of subclusters.

Currently I'm using scikit-learn AgglomerativeClustering. I looked through a question related to mine.

I found that the merging operation for max link and average link happens in the script.

However, I cannot find where the the distances between all the subclusters are calculated and how to derive a number of edges between leaves and roots in subclusters.

How can this be done?

UPD: I am trying to implement Chakrabarti algorithms for smooth clustering over time. For agglomerative clustering, a choice for next clusters to be merged depends on two components: similarity between subclusters at current time stamp and a historical cost related to the previous time stamp. The latter has several variations. For example, one includes merge distance (not only that): if we would merge two subclusters S1 and S2, what would be the average distance (in terms of edges) between all leaves in S1 and S2.

There are a couple of questions about sklearn AgglomerativeClustering:

What lines in the code implement a choice of clusters to be merged at the current step?
Is it possible to extract a distance between a leaf and a root of a subcluster at each merging step? I noticed there is an attribute 'children_' in AgglomerativeClustering which I could use to create the resulting tree and find distances between any nodes. How to extract such an information for every merging step?


texts: Is it correct to argue that the Bayer Classifier is an ideal classifier, which is taken as a model by every other implemented classifier?


texts: I am currently working on a project where my job is to intent analysis of an article. Suppose i am given with the article and i want to classify that what kind of article it is?. Any suggestions, Which kind of model would be best to apply?


texts: How does inverse of logistic function produces "linear relationship", (so we can use least-squares)

texts:
I am reading about time-series analysis in "A First Course on
Time Series Analysis".  The book reviews the logistic function ($f_{log}(t)$).

Part 1.6 (PDF page 16; book page 8; screenshot below) explains how the inverse of the logistic function ($1/f_{log}(t)$) produces a linear relationship; so that "it can serve as a basis for estimating the parameters $β_1$, $β_2$, $β_3$ by an appropriate linear least squares approach"

I don't understand the last sentence:

  This means that there is a linear relationship among $1/f_{log}(t)$.

I specifically don't understand how the inverse of a logistic function, can produce a "linear relationship"; it doesn't look like a "straight line" when I graph it:

import math
import numpy as np
from matplotlib import pyplot as plt

get_logistic = lambda b1, b2, b3: lambda x: (b3/(1+b2*np.exp(-b1*x)))

x = np.linspace(-10,10)
y = get_logistic(1,1,1)(x)

plt.plot(x,y)
plt.show()
plt.plot(x,1/y)
plt.show()


texts: AttributeError: type object 'Lambda' has no attribute 'shape'

texts:
I'm working on a sequence2sequence model with attention mechanism, thus I'm using an encoder-decoder architecture.

The decoder part code is :

# Now create the Decoder layers.
decoder_inputs = Input(shape=(len_target,))
decoder_emb = Embedding(input_dim=vocab_out_size, output_dim=embedding_dim)
decoder_lstm = LSTM(units=units, return_sequences=True, return_state=True)
decoder_lstm_out, _, _ = decoder_lstm(decoder_emb(decoder_inputs),
               initial_state=encoder_states)
x2 = reduce_dim(decoder_lstm_out)
attn_layer = AttentionLayer(name='attention_layer')
attn_out, attn_states = attn_layer([z, x2])
decoder_concat_input = Concatenate(axis=-1, name='concat_layer')([x2,
               attn_out])
lm= Lambda(decoder_concat_input)
decoder_d2 = Dense(vocab_out_size, activation="softmax")
decoder_out = decoder_d2(lm)
model = tf.keras.Model([encoder_inputs, decoder_inputs], decoder_out)

The reduce_dim function() is:

   def reduce_dim(decoder_lstm_out):
       x = decoder_lstm_out
       x = tf.placeholder(tf.float32, shape=[None, 12, 12, 256])
       dim = tf.reduce_prod(tf.shape(x)[0:1])
       x2 = tf.reshape(x, [dim,12,256])
       return x2

An error is raised :

  AttributeError: 'Lambda' object has no attribute 'shape'

At the line:

  decoder_out = decoder_d2(lm)

If anyone can help me please?


texts: 100% Accuracy on test dataset using a previous developed model oputput

texts: My dependent variable is a probability that is sourced from someone else's classification model. I am using this probability as a dependent variable as I don't have the actual data. On building an xgboost algorithm, the accuracy is 100%. There is no data leakage - but I wanted to ask - is a 100% accuracy possible when modeling a previously developed algorithm?


texts: How to build a comunication interface between a reinforcement learning and a videogame?

texts:
I want to try to build a reinforcement learning model in order to play old arcade videogame, I know about projects in order to build AI for playing videogames, but I don't know how could I build an interface between a videogame and a python app in order to train the model while the model is playing.
is there any library in order to train a reinforcement learning model using an emulator or something like that on python? How can I build an app like that?
Thanks


texts: How to replace NA values with another value in factors in R?

texts:
I have a factor variable in my data frame with  values where in the original CSV "NA" was intended to mean simply "None", not missing data. Hence I want replace every  value in the given column with "None" factor value. I tried this:

DF$col[is.na(DF$col)] <- "None"

but this throws the following error:

Warning message:
In `[<-.factor`(`*tmp*`, is.na(DF$col), value = c(NA, NA,  :
  invalid factor level, NA generated

I guess this is because originally there is no "None" factor level in the column, but is it the true reason? If so, how could I add a new "None" level to the factor?

(In case you would ask why didn't I convert NAs into "None" in the read.csv phase: in other columns NA really does mean missing data).


texts: Mask R-CNN Background Subtraction Implementation

texts: I am currently attempting to reimplement a paper on fall detection (https://ieeexplore.ieee.org/abstract/document/9186597). It requires a background subtraction algorithm called Mask R-CNN. Are there any current implementations of this algorithm for background subtraction?


texts: What do mean and variance mean for high dimensional data?

texts:
I'm using Scikit Learn to guess the tag of Stack Overflow posts given the title and body. I represent the title and body as two 300-dimensional vectors of floats. The documentation for Scikit learn's SGDClassifier states:

  For best results using the default learning rate schedule, the data should have zero mean and unit variance.

With one-dimensional data, I understand what mean and variance are but when we go into higher dimensions how would one calculate the mean and variance for a dataset?


texts: What kind of insights can we give from point A to point B transactions. A and B are two places, All I can think of is about descriptive statistics like how many transactions from A and what is count of transactions between places and all.


texts:
Is it necessary to append train X & test X with .values? But I checked and found the model works fine without appending .values, then why append?

In other words.. which among the following is better, and why?

X = df.iloc[:,4].values

OR

X = df.iloc[:,4]


texts: Data set compatability with LSTM's or convLSTM's

texts:
I am currently working on a data set which has the structure - frames x Number of Objects.
Each object is of size 7x5 and the task is to classify each object into one of the 4 classes.
I have been getting good results using CNN but we also have a temporal component in our data and hence want to try out convLSTM.
The Problem - Each frame in the data set has a timestamp and is temporally in sequence but the objects within a frame are not related in the time domain. How could I restructure my data so that I could use convLSTM/LSTM and predict class for each object? Basically I want to explore temporal relationship between predictions of two consecutive frames instead of looking at predictions of two consecutive objects.
I would appreciate any help/suggestion/tip.


texts: What is the minimum number of times a word needs to appear in word2vec training corpus for quality results?

texts:
When training a word2vec model with, eg, gensim, you can specify the minimum times a word needs to be seen (with the parameter min_count). The default value for this seems to be 5.

Are there any theoretical considerations for selecting a threshold for min_value? Depending on the contexts the words are seen in (and the potential variety of them), it seems that it could take more than 5 words for the model to learn a good vector representation of the word. Are there any papers that specify when the learned vectors become more static? Alternatively, can you train the model on all words, but only utilize the vectors of words that appeared in the corpus more than a specified number of times? Is frequency a better cutoff than min_count?


texts: How to print x-axes labels in pandas.Series.plot()?

texts:
I am trying to visualise my data to understand the data skewness. For that purpose, I use the below and get desired output -

df.groupby('owner_team').inc_subj.count().plot.bar(ylim=0)

Output -

My concern is the x-axes labels are shown as numbers, which is the exactly values present. But, my desire is to see the names (string values) corresponding to those numbers.

To give a little bit of background, initially they were string values which I converted to integer values using factorize():

df['owner_team'], mapp = df['owner_team'].factorize()

I am referencing this Pandas doc but couldn't find the exact parameter to set.
Tried labels but didn't help.

Any pointers please.

ps. Using Pandas v0.23.4 and Python v3.6


texts: What's the best way to train a model when having a very big dataset split in batches?

texts:
I was wondering what is the best way to train my Keras Model, in a binary classification context.
My full dataset is composed of 2.3M rows and 120 columns, which is really big.  As you would imagine, it doesn't fit into memory. To avoid memory errors, I have split this big dataset into 23 CSVs of 100.000 rows each. I have shuffled the full dataset prior to the splitting, so we have 23 i.i.d. samples.
I have two options:

Take the first CSV, train the model with say 20 epochs (with mini-batches of 256), then take the next CSV, and train the same model with 20 more epochs, and so on and so on 23 times...
Create a Python Generator that yields batches from the 23 CSVs in a way that at the end of the first epoch, the model has seen the whole 2,3M datapoints.

The main difference is the order in which the model sees the data. In both methods, the model will have seen 20 times the whole dataset.

In the first method, the model sees 20 times the same 100.000 datapoints, then moves to the next step.
In the second method, the model sees 1 time the whole dataset, then moves to the next step.

The reason I am asking this is because the first method is way easier to implement. A simple for loop can do the work. The second method requires Python Generators and mini-batch sampling from the already sampled CSVs.
Which of those way is the best to train my model ?


texts: linear mixed modelling- Zi in model matrix

texts:
Hi I am trying to learn linear mixed modelling. In the matrix form of the model $Y_i = X_i \beta + Z_i \mu_i + \epsilon_i$, how do we calculate is $Z_i$? The model is defined in the picture attached.

Thanks!


texts: hierarchical clustering doesn't work as expected

texts:
I have a precomputed distance matrix.

I'm trying to do an hierarchical clustering using scipy:

from scipy.cluster.hierarchy import fcluster, linkage, cut_tree,fcluster

z = linkage(ds, "average")
clust = fcluster(z, 4, criterion='maxclust')
print(clust)

The problem is that, most of the labels ends up in one group - Why?

By the way, I tried to use AgglomerativeClustering as well but I got similar results.

ds is the precomputed distance matrix. I'm using DTW, a distance metric for time series analysis.

I uploaded ds.csv to here.

Also, here's an heatmap which forms interesting lines (I don't really know how to interpret this currently)


texts: What is dropout in convolutional layers and how does that different from max-pooling-dropout?

texts:
When dropout is applied to fully connected layers some nodes will be randomly set to 0.

It is unclear to me how dropout work with convolutional layers. If dropout is applied before the convolutions, are some nodes of the input set to zero? If that so how does this differ from  max-pooling-dropout? Even in max-pooling-dropout some elements in the input are randomly dropped (Set to zero).


texts: Why doesn't class weight resolve the imbalanced classification problem?

texts: I know that in imbalanced classification, the classifier tends to predict all the test labels as larger class label, but if we use class weight in loss function, it would be reasonable to expect the problem to be solved. So why we need some approaches like down sampling or up sampling for imbalanced classification problem?


texts: Using pre-trained Faster R-CNN models in Mask R-CNN

texts:
I am planning to use mask r-CNN from TensorFlow Object Detection API for one of my projects.
When I checked the TensorFlow 1 Detection Model Zoo, I found that there is faster_rcnn_inception_resnet_v2_atrous_oidv2 and faster_rcnn_inception_resnet_v2_atrous_oidv4 but no Open Images-trained Mask R-CNN models present in there.
That made me curious, is it possible to use pre-trained Faster R-CNN models for transfer learning in Mask R-CNN. So, I don't have to take the hassle of training a model from scratch using the Open Images dataset.


texts: I would like to ask how to set paramater k in function SelectKBest for feature selection. I have now around 2300 features, so I think that default value 10 is not enough. Is there any approach, how many features choose or it is just on testing to find some compromise between accuracy and number of features?


texts: Increase accuracy of classification problem

texts:
I am trying to build a classifier that predicts the compiler given some operations of assembly code. Here is the pandas dataframe:

What I do is using a TfidfVectorizer and select the features that have most predictive power by doing:

tfidf_vectorizer=TfidfVectorizer(max_features=500)

so using max_features=500 to select the 500 features with the highest idf. The problem is that the accuracy is still low, infact it is around 0.69. I would like to arrive at least at 0.9, but I dont't know what else do.

I am using support vector machines and this gives me the accuracy of 0.69. I also tried random fores and I was around 0.75.

My code is the following:

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer=TfidfVectorizer(max_features=1000)
df_x = df['opcodes']

X_all = tfidf_vectorizer_vectors=tfidf_vectorizer.fit_transform(df_x)
y_all = df['compiler']

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all,
      test_size=0.2, random_state=15)

from sklearn import svm

model = svm.SVC(kernel='linear', C=1).fit(X_train,y_train)

y_pred = model.predict(X_test)

acc = model.score(X_test, y_test)
print("Accuracy %.3f" %acc)

moreover the dataset is balanced, infact I have 3 compilers with 1000 samples each.

I don't know what other startegy to try to increase accuracy and get at 0.9.

Can somebody please help me?  Thank's in advance.


texts:
I'm reading the cornerstone paper Sequence to Sequence Learning
with Neural Networks by Ilya Sutskever and Quoc Le. On the first page, it briefly mentions that:

A surprising example of the power of DNNs is their ability to sort
N N-bit numbers using only 2 hidden layers of quadratic size

Can anyone briefly outline how to sort numbers using only 2 hidden layers?


texts: Where is $SPARK_HOME on an EC2 instance?

texts: I am working on an EC2 instance with anaconda and pyspark pre-installed. My $SPARK_HOME variable is not set. Any ideas on how to locate my spark home?


texts: K-Nearest Neighbours algorithm explanation needed

texts:
I need some explanation for K-Nearest Neighbors algorithm.

Why is the training process needed in KNN algorithm? In regression
models the training process means to find optimum parameters for a
selected function. Does the training in KNN mean that the similarity
distances are calculated between training datapoints? Why is it good
for a later prediction phase?
Are similarity distances also calculated between training and
testing datapoints? If so then why is it necessary at all that
before that the distances between training datapoints were
calculated?
If the following code is executed then what does it exactly do? Does
it mean that distances between training datapoints are calculated?

neigh = KNeighborsClassifier(n_neighbors = k).fit(x_train, y_train)
If the following code is executed then what does it exactly do? The
manual says that class labels for each data sample are returned. But
how is it done? Does it mean that distances between training and
testing datapoints are first calculated? If not then how can we find
neighbors for "unseen datapoints" at all? We need to know distances.

yhat = neigh.predict(x_test)


texts: How is DS used in the case of Payment Gateways?

texts:
I know it's a general question but what type of analytics can be done in this case?
How can we apply machine learning models here?


texts: metric learning and information retrieval

texts:
I am interested in parsing semi-structured text.
Assume that I have a text with labels of the kind: year_field, year_value, identity_field, identity_value, ..., address_field, address_value, and so on.

These fields and their associated values can be everywhere in the text, but usually they are near to each other, and more generally the text in organized in a (very) rough matrix, but rather often the value is just after the associated field with eventually some non-interesting information in between.

The number of different format can be up to several dozens, and is not that rigid (do not count on spacing, moreover some information can be added and removed).

I am looking toward machine learning techniques to extract all those (field,value) of interest.

I think metric learning and/or conditional random fields (CRF) could be of a great help, but I have not practical experience with them.

Does anyone have already encounter a similar problem?

Any suggestion or literature on this topic?


texts: Reinforcement Learning - PPO: Why do so many implementations calculate the returns using the GAE? (Mathematical reason)

texts:
There are so many PPO implementations that use GAE and do the following:
def compute_gae(next_value, rewards, masks, values, gamma=0.99, tau=0.95):
    values = values + [next_value]
    gae = 0
    returns = []
    for step in reversed(range(len(rewards))):
        delta = rewards[step] + gamma * values[step + 1] * masks[step] - values[step]
        gae = delta + gamma * tau * masks[step] * gae
        returns.insert(0, gae + values[step])
    return returns

...
advantage = returns - values

...
critic_loss = (returns - value).pow(2).mean()

Soure: https://github.com/higgsfield/RL-Adventure-2/blob/master/3.ppo.ipynb, I made only slight changes for better readability.

I understand why we use the function to calculate the advantage.  But why are we using the same formula to also calculate the target values for the value network? The value network should output the value function. And wouldn't it be sufficient to just use discounted rewards to calculate the value function? E.g. why do we need $\lambda$ here? Shouldn't the target values be independent of $\lambda$?
In the code they call it returns. What would be the mathematical symbol for it? I mean it is probably not $R_t$, right?
Are the "returns" we are calculating here the same as:

$$\hat{V}_t^{GAE} = (1 - \lambda) \sum\limits_{N > 0} \lambda^{N - 1} \hat{V}_t^{(N)} \approx V^{\pi}(s_t)$$
(Source: Paper What Matters In On-Policy Reinforcement Learning?
A Large-Scale Empirical Study) And if that is the case then we have indeed a different target function as usual, right? Would it also work with just discounted rewards? And what is the reason why are people are instead using this calculation?
So my basic question is what is the (deep) mathematical reason why we are calculating the target values for the value network in this way?
Thank you very much in advance for your help!


texts: Why convolutions always use odd-numbers as filter_size

texts:
If we have a look to 90-99% of the papers published using a CNN (ConvNet).
The vast majority of them use filter size of odd numbers:{1, 3, 5, 7} for the most used.

This situation can lead to some problem: With these filter sizes, usually the convolution operation is not perfect with a padding of 2 (common padding) and some edges of the input_field get lost in the process...

Question1: Why using only odd_numbers for convolutions filter sizes ?

Question2: Is it actually a problem to omit a small part of the input_field during the convolution ? Why so/not ?


texts: How to properly vectorise when I have several text features?

texts:
I am having several features as text and I'm using them once for a classification problem and then later for a regression problem.

The textual data features themselves aren't categorical. Because as far I understand even though we generally classify features as either numerical or categorical but the text features do not have a limited set of values.

So I have two questions mainly:

There are several NaN values, so should I really replace them with the most frequent value or replace by an empty string? The most frequent value occurs some 200 times out of a total no of values of 20k in each of the three features.
I initially used TfidfTransformer and TfidfVectoriser to convert each feature to vector. However, each vector became very big in size. In figures of 19k - 30k. Each vector of different sizes. Then later I used HashVectoriser and converted each to a vector size of 1000 vectors. However, my classification problem still had a very low accuracy. So how do I identify what is the ideal size of vectors? Or should i try doing something else like combining the three features of each row and then vectorise them?

I'm kind of new to this, so thank you!


texts: Is there away to change the metric used by the Early Stopping callback in Keras?

texts:
When using the early stopping callback in Keras, training stops when some metric (usually validation loss) is not increasing. Is there a way to use another metric (like precision, recall, or f-measure) instead of validation loss?
All the examples I have seen so far are similar to this one:
callbacks.EarlyStopping(monitor='val_loss', patience=5, verbose=0, mode='auto')


texts: How to check the similarity between two transition matrixes

texts:
I have two transition matrixes in which the probabilities for the transition between each from state to each to states are recorded. For instance

trans_mat1 is

      s1    s2    s3
s1    0.23  0.15  0.26
s2    0.82  0.01  0.01
s3    0.27  0.78  0.08

trans_mat2 is

      s1    s2    s3
s1    0.05  0.66  0.44
s2    0.57  0.70  0.12
s3    0.71  0.01  0.28

These are dummy transition matrixes to show an MWE. I am confused how to check if these two transition matrixes show the same transition pribabilities given that s1, s2, and s3 are same in both matrixes?

Can I treat them as a single probability distribution vector and calculate KL divergence using scipy.stats.entropy function ?


texts:
I have a lot (thousands are possible)  of automatically-generated ordinal features that i'd like to exploit , to differentiate between two classes. I'm looking for some measure that will select the most-likely-to-be-useful ones .  For instance I could look at the frequency of occurrence of each feature for one class vs. the other class, and go for those features that have the greatest delta .
If the features are ngrams from given names, and the classes are latin-american vs. american names , the frequency histograms look like this : latin ngrams,
english ngrams

Is there some better way to go about this than looking for large frequency deltas? The scikit-learn classifiers allow you to check for feature usefulness post-facto, I was looking to do this pre-classifier.


texts:
I provide an offline library of Music to my users.
My goal is to understand what my users are looking for, which means translate raw user searches to: Music Artists, Songs, Albums and then add music to the company library.

What are the suggested clustering algorithms to group common short sentences into a single entity. Example:

Taylor Swift Shake it Off
Taylor Swift Shaek it Off
Shake it off
Twylor Swift Shake it Off

I tried this example and works fine for a specific number of clusters (K) where K < N | K <= N. But since searches are unpredictable need to find a way to automate the number of clusters: my goal is to cluster 2 or more similar items and let alone single searches in independent clusters, example:

Cluster 1:

Taylor Swift Shake it Off
Taylor Swift Shaek it Off
Shake it off
Twylor Swift Shake it Off

Cluster 2

Avicci
Avicii Ibiza
Avicci Electro House

Cluster 3

Juan Gabriel

Cluster 4

Adele

Cluster 5

Britney Spears
Britney Spears VMA 2016


texts: Handling different length string features and prediction of these based on other features

texts:
I am currently working on a problem where the dataset contains 200+ features (Let's call them the code features, e.g no.of.loops, memoryInst, loadInst, etc  and Flags that are used to compile code that has such characteristics/code features)

The flags are represented as strings:

This is just dummy data.

snippet                 FlagsUsed                   no.of.loops   loadInst      memoryInst

1  Mergesort      " -a -b -c -d=10 -e -f =19  -c "    1              0             10

2  Bubblesort     " -a -c -f=230 "                    2              5             3

3  MatrixMulti     " -f=20 -z -f12 -f2f "             0              10            4

I need some help with how these flags should be represented in the data, I have tried one-hot encoding and dummy_variable methods but these methods have some disadvantage:

1) One-hot encoding: using this method, the no. features will become huge, there are 150+ flags and I creating a one hot vector for each one of them would result in 22500 feature which is not feasible.

   flag1    flag2    flag3  flag4
1  -a       -b       -c      NA
2  -c       -a       -b      -z

I would need to create long vectors for each of the above feature flag1, flag2.. flag4

2) Dummy_variable method: There are 200+ different types/levels/factors of these flags and dummy_variable method create a feature for each level/factor which is not feasible.

Also, the flags can repeat in a single string (1st snippet, -c repeats).

I am thinking of some clever hashing that would maintain information regarding the sequence of the flags and there value (toggle flag = 0/1, threshold flag = {lower, upper} ). But the problem with hashing is, I have to, in future, predict these flags using other features (code features) and if I hash these flags somehow I won't be able to reverse hash them.

I am thinking of some fixed size vector representation which could be reversed so that I can tell flag using a numeric or hex number.

Can anyone please guide me or put me in the right direction. Would be thankful!


texts: neural network for binary classification of xor gate

texts:
i have written this neural network for XOR function.the output is not correct.it is not classifying the test inputs correctly.can anyone please let me the reason why.

import numpy as np
import pandas as pd
x=np.array([[0,0],[0,1],[1,0],[1,1]])
y=np.array([[0],[1],[1],[0]])
np.random.seed(0)
theta1=np.random.rand(2,8)
theta2=np.random.rand(8,1)
np.random.seed(0)
b1=np.random.rand(4,8)
b2=np.random.rand(4,1)
alpha=0.01
lamda=0.01

for i in range(1,2000):
    z1=x.dot(theta1)+b1

    h1=1/(1+np.exp(-z1))
    z2=h1.dot(theta2)+b2
    h2=1/(1+np.exp(-z2))

    dh2=h2-y
    #back prop

    dz2=dh2*(1-dh2)
    H1=np.transpose(h1)
    dw2=np.dot(H1,dz2)
    db2=np.sum(dz2)

    W2=np.transpose(theta2)
    dh1=np.dot(dz2,W2)
    dz1=dh1*(1-dh1)

    X=np.transpose(x)
    dw1=np.dot(X,dz1)

    db1=np.sum(dz1)

    dw2=dw2-lamda*theta2

    dw1=dw1-lamda*theta1
    theta1=theta1-alpha*dw1
    theta2=theta2-alpha*dw2
    b1+=-alpha*db1

    b2+=-alpha*db2

#prediction
#test inputs
input1=np.array([[0,0],[1,1],[0,1],[1,0]])
z1=np.dot(input1,theta1)

h1=1/(1+np.exp(-z1))
z2=np.dot(h1,theta2)

h2=1/(1+np.exp(-z2))

expected output=[0],[0],[1],[1]
actual output=[[ 0.95678049]
 [ 0.99437206]
 [ 0.98686979]
 [ 0.98628204]]

all are ones here.


texts: scikit-learn n_jobs parameter on CPU usage & memory

texts:
In most estimators on scikit-learn, there is an n_jobs parameter in fit/predict methods for creating parallel jobs using joblib.
I noticed that setting it to -1 creates just 1 Python process and maxes out the cores, causing CPU usage to hit 2500 % on top.
This is quite different from setting it to some positive integer >1, which creates multiple Python processes at ~100 % usage.

How does setting it affects CPU & core usage on a multi-CPU Linux server?
(e.g. if n_jobs=8 then are 8 CPUs fully locked up or do the CPUs still reserve some cores for other tasks/processes?)

Additionally, I do get MemoryError occasionally when setting n_jobs=-1 for large datasets.
However, the memory usage usually hovers at around 30-40 % for the single Python process.
How is the data & memory being managed/copied depending on the value of n_jobs?


texts: Python - Get FP/TP from Confusion Matrix using a List

texts:
I using two different classifiers to predict a binary target (Random Forests and Decision Trees). Now I want to evaluate my model creating a confusion matrix. For example, for predicting the binary value using random forests I've:

training_features, test_features, training_target, test_target, = train_test_split(df.drop(['score_goal'], axis=1),
                                              df['score_goal'],
                                              test_size = .3,
                                              random_state=12)

clf_rf = RandomForestClassifier(n_estimators=25, random_state=12)
clf_rf.fit(training_features, training_target)
print("Accuracy using Random Forest Classifier is ", clf_rf.score(test_features, test_target)*100)

I'm confusing because I don't know how I can compare the predicted values to identify how many False Positives, etc. I have.

Anyone knows how can I build that function?

Thanks!


texts: Can I apply Hidden Markov Models this way?

texts:
I have just gotten my feet wet with Hidden Markov Models. Now I want to apply them to tell whether a transaction from an ATM is suspicious or not. I have great confusion in defining my Hidden States.

My attempt: A transaction is either suspicious or non-suspicious. Since I cannot tell, ("this is hidden to me"), whether a transaction is suspicious or not before analyzing it, therefore my Hidden States(Suspicious, Non-Suspicious). Does this hold?

My great confusion: When I read about Hidden Markov Models on Weather example, I understood it this way: "weather can be rainy or sunny - I mean its the 'same' weather that is switching between the two states". In my transaction case - one transaction cannot switch between being suspicious and non-suspicious. So, does my hidden states hold?

Questions: Does my hidden states hold? Can I apply the HMM that way? Please advice, thanks.

P.S: I had my Observations as location, time and amount of money but as of now I want clarification on Hidden states.


texts: Stratified K fold same index present in both test and valid set

texts:
I  am trying to do a stratified k fold cross validation for my dataset and want to keep an isolated 10% test set from the dataset and use the remaining for training and validation. Below is the code which I tried, but getting a few indexes of test index common in both train and valid set. Ideally this should not be happening as I have separated the test set in the beginning.

# Splitting into test and tmp set , tmp set to be further split into train and
valid

tmp_idX, test_idX = train_test_split(idx,test_size=0.1,random_state=2)
#idx is an array with number of files, eg for 100 files idx has elements 0-99

tmp_Y = np.asarray(grps.iloc[tmp_idX, 0])

skf = StratifiedKFold(n_splits=10, shuffle=True, random_state=1) # do 10 fold
xval

for train_idX,valid_idX in skf.split(tmp_idX,tmp_Y):

  #Define model and fit model

Please let me know where I am going wrong..

Thanks in advance for your reply


texts: What are the cases in which Isomap fails to do a good job?

texts: As above, what is a possible scenario/ dataset/ case in which Isomap fails to do a decent dimensionality reduction?


texts: Actions taken by agentn/ agent performance not improving

texts:
Hi I am trying to develop an rl agent using PPO algorithm. My agent takes an action(CFM) to maintain a state variable called RAT in between 24 to 24.5. I am using PPO algorithm of stable-baselines library to train my agent.I have trained the agent for 2M steps.

Hyper-parameters in the code:

def __init__(self, *args, **kwargs):
        super(CustomPolicy, self).__init__(*args, **kwargs,
                                           net_arch=[dict(pi=[64, 64],
                                                          vf=[64, 64])],
                                           feature_extraction="mlp")

model = PPO2(CustomPolicy,env,gamma=0.8, n_steps=132, ent_coef=0.01,
             learning_rate=1e-3, vf_coef=0.5, max_grad_norm=0.5, lam=0.95,
             nminibatches=4, noptepochs=4, cliprange=0.2, cliprange_vf=None,
             verbose=0, tensorboard_log="./20_01_2020_logs/", _init_setup_model=True,
             policy_kwargs=None, full_tensorboard_log=False)

Once I train the agent I am testing the actions taken by agent in an episode.

For the time steps in between 40 and 60 ,RAT value is higher than 24.5. From the domain knowledge, if agent takes an action of CFM around 250, it can maintain the RAT in between 24 to 24.5. But agent is not taking such actions instead taking actions similar to previous steps.

Can someone help me with how to tackle this problem? Is there any particular hyper parameter I should try tuning?

Thanks


texts: How to identify the appropriate predictors in regression model

texts:
I have a big data and want to perform some of prediction models(such as Regression,Decision Tree,..) on it in R but I have a problem. As there are too many variables, Is there any way to identify which variables should i use as predictors in the model to predict a specific variable?
Any help would be greatly appreciated.


texts: I'm facing the following problem of integrating data from another company  (data base) to an internal one. It's about personal core data, i.e. name address etc. I would like to come up with a mapping of these keys in an automated way. I've read about HMM to use for this. However, I'm still gathering some information of some feasible and standard ways to do it. I'm looking therefore for some reference which describes a possible solution to this.


texts: Feature extraction of accelerometer data for machine learning

texts:
Currently, I am doing a project with the aim of classifying potholes through machine learning. The data collected is from an accelerometer in which the z-axis measures the "vertical" acceleration of the car, when a pothole is struck.

I have tried to deconstruct the signals and create features using two methods:

PACF along with a moving average to combat the noise
Calculated a periodogram for spectral analysis

Is there any other ways to differentiate a pothole from "rough" road surfaces, as I am unsure which approach to take?


texts: questions on multi-step air pollution prediction

texts:
I am trying to use RNN to predict the concentration of various air pollutants for the next 24 hours.

The input data consists of 72 hours long and every hour owns 14 elements such as temperature, windspeed, and the history concentration of several pollutants. The output is the concentrate of a certain pollutant in the next hours.
However, I found that if I used LSTM to predict just one-future-hour concentration, I could get a high accuracy on it. when I try to do prediction in a long term such as a 24-future-hour prediction, I choose to use an encoder-decoder framework to predict future 24 hours concentration.  The problem is that no matter how I adjust the model, I cannot get as accurate as a single LSTM in the first hour, but a multi-step model is hard to do a rolling forecast by predicting all the input elements. (to be honest, I did it and it really was more accurate than an encoder-decoder framework in the first few hours, and then it is less accurate than the encoder-decoder one) I do some survey on some paper and they said it's better to cut the 24 hours into many segments. This method can promote accuracy in every hour.
I want to know that why it is and I wonder that is there any model that can deal with this problem without cutting itself into several segments? Or am I wrong in understanding the model I used and didn't unleash its potential?


texts:
Looks like Pandas doesn't translate Pandas timedelta to Parquet INTERVAL:
>>> import pandas as pd
>>> df = pd.DataFrame([{'seconds': 30}])
>>> df.to_parquet('/tmp/test.parquet') # so far so good
>>> df['duration'] = pd.to_timedelta(df.seconds, unit='seconds')
>>> df.to_parquet('/tmp/test.parquet')
Traceback (most recent call last):
  / ... /
pyarrow.lib.ArrowNotImplementedError: Unhandled type for Arrow to Parquet schema conversion: duration[ns]
>>>

Is this simply a missing feature? Am I wrong to expect timedelta to be saved as INTERVAL? Which format would you recommend if my dataframe is quite large (500mb) but I'll read it back to Pandas - .to_pickle()?


texts: How to utilize dictionary data set for text classification?

texts: I have a dataset similar to newsgroup20 for classification. With the training dataset, I have a dictionary data set that explains some jargons in the training dataset. These both are different data set, So how will i utilize the dictionary dataset for improving my model accuracy?


texts: Why has Hadoop failed to become popular?

texts:
Apache Hadoop was once considered one of the tools you should have as a Data Scientist. Around 2012 to 2014, it gained the popularity steeply among Data Scientists, and seemed to be considered one of the toolsets on the same boat as Python, R, and SQL.

But these days I have not heard its story. In fact, the Google Trend also agrees with me, that the popularity is peak at May, 2015.

Why has Hadoop failed to become one of the necessary tools as a Data Scientist, like Python or R?


texts: Topic modeling for short length sentences

texts:
I have a graph which was already separated into clusters. Each node in the graph has a label (typically, it's a function's name like org.java.somepackage.validateLogin). What I want to do is to give a representative label for each cluster.

For the sake of simplicity, let's assume I'm able to clean the data (i.e. break 'validateLogin' to 'validate' and 'login')

I've done a little research on the subjects of topics modeling and cluster labeling and encountered a few algorithms, such as: LDA, NMF and TF-IDF (not quite an algorithm on itself)

Basically, many algorithms are documents oriented which are reach of words and not short-text/labels oriented.

It is worth mentioning:

We can use the fact that different clusters might have different labels, so maybe a proper label for a cluster could be unique words in the overall bag of words (that can be done with TF-IDF I guess)
Labels can be one word but can also imply a hierarchy (i.e. packageA.packageB.packageC.funcName)

I'd be glad if you could give me your insights on this problem and what approach could fit here.


texts: What is the best deep learning library for scala?

texts: Does any one has a recommendation for what libraries to use for deep learning?


texts: ValueError: X has 3 features per sample; expecting 1500

texts:
import numpy as np
from flask import Flask, request, jsonify, render_template
import pickle
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import LogisticRegression
from flask_bootstrap import Bootstrap
app = Flask(__name__)
Bootstrap(app)
model = pickle.load(open('model.pkl', 'rb'))

@app.route('/')
def home():
    return render_template('index.html')

@app.route('/predict',methods=['POST'])
def predict():
    '''
    For rendering results on HTML GUI
    '''
    vectorizer = TfidfVectorizer()
    namequery = request.form['namequery']
    data = [namequery]
    vect = vectorizer.fit_transform(data)
    my_prediction = model.predict(vect)
    return render_template('result.html',prediction = my_prediction)

if __name__ == "__main__":
    app.run(debug=True)

here is preprocessing of dataset code

import numpy as np
import pandas as pd
import re
import string
import nltk
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
import matplotlib.pyplot as plt
import seaborn as sns
import pickle

data = pd.read_csv('train.csv')
#removing id column
data=data.drop('id',axis=1)
string.punctuation
nltk.download('stopwords')
stopword=nltk.corpus.stopwords.words('english')
def remove_punct(text):
  text_nopunct=''.join([char for char in text if char not in string.punctuation])
  return [word for word in text_nopunct.split() if word.lower() not in stopword]
x=data['text'].apply(remove_punct)
ps=nltk.PorterStemmer()
def stemming(text):
  text=[ps.stem(word)for word in text]
  return ' '.join(text)
x=x.apply(stemming)
vectorizer = TfidfVectorizer(max_features=1500)
X = vectorizer.fit_transform(x)
print(vectorizer.get_feature_names())
y=data['target']
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.25,random_state=0)
# instantiate the model (using the default parameters)
logreg = LogisticRegression()

# fit the model with data
logreg.fit(X_train,y_train)

#
y_pred=logreg.predict(X_test)
from sklearn import metrics
cnf_matrix = metrics.confusion_matrix(y_test, y_pred)
cnf_matrix
class_names=[0,1] # name  of classes
fig, ax = plt.subplots()
tick_marks = np.arange(len(class_names))
plt.xticks(tick_marks, class_names)
plt.yticks(tick_marks, class_names)
# create heatmap
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, cmap="YlGnBu" ,fmt='g')
ax.xaxis.set_label_position("top")
plt.tight_layout()
plt.title('Confusion matrix', y=1.1)
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
# Saving model to disk
pickle.dump(logreg, open('model.pkl','wb'))

# Loading model to compare the results
model = pickle.load(open('model.pkl','rb'))

please help me someone to reslove this error. Iam new to machine learning am not understanding where i made mistake.
Thanks in advance!!


texts: Why is it a problem that your response variable is skewed in regression? Is taking logarithms the only way to solve it?


texts:
I have no idea whether this is the right StackExchange flavor to post this question in, but here goes:

I have an agent (an IRC bot that listens to an event stream) that I would like to add "intelligent notifications" to. Let me give a specific example.

The bot monitors the results from our continuous integration system, so it basically sees a stream of test results associated with changes (and the changes are associated with users.) If I ask it to notify me of interesting events, then for each event it sees, it decides whether to tell me about it (or rather, about the current results so far.)

I would like it to decide based on how interesting something is for me. Something is interesting if (1) it is a result of a change that I pushed to the continuous integration system, (2) it is a test failure, and (3) the information conveyed by that failure is a significant indicator of whether my change was bad.

Tests have a baseline probability of failing for no good reason ("intermittent failures"). If one test fails, it does not necessarily mean my change was bad. It might just be a flaky test. (Any given push tends to result in a couple bogus failures, so this isn't some obscure edge case.) But the baseline probability of an intermittent failure varies according to the test suite running (we have several dozen different test suites that fire off for every change pushed.)

So I don't want to be bothered with reports of failing tests that are probably just noise. And if my push is bad, I don't want to be flooded with notifications for every failed test. (If I break something, it'll probably show up in multiple test suites. So if the agent sees two failures and 20 successes, maybe it won't bother me, but if it then sees another failure or two it should.)

I am imagining I could look at the change between the prior and posterior probabilities and use that to estimate the entropy of that new test result, and perhaps also compute my personal entropy (if that makes any sense -- as in, the bot should assume I don't know what any of the results are until it notifies me, at which point it should assume I am aware of the current full set of results and not bother me again until enough additional results have come in to substantially change the probability estimate of my change being bad.)

Or something like that. I'm really looking for the right mathematical framework to compute things like this. I know next to nothing about machine learning or... well, mathematics in general.

(The above is simplified; I would additionally like to do crazy things like take into account "labeling", where a third party looks at the test failures and decides whether they are real failures or not. But the time between the failure coming in and when that person labels it for me can also be modeled by a distribution, and I'd like to hold off notifications for interesting results if there's a good chance that this other person may tell me that the result is not interesting after all. But only if that's going to happen "soon". Also, we have an automated system for guessing whether something might be an intermittent vs real failure, and it'd be nice to take its opinion into account as well.)


texts: Always drop the first column after performing One Hot Encoding?

texts: Since one of the columns can be generated completely from the others, and hence retaining this extra column does not add any new information for the modelling process, would it be good practice to always drop the first column after performing One Hot encoding, regardless of the algorithm of choice ?


texts: What is deconvolution operation used in Fully Convolutional Neural Networks?

texts:
When I was reading this this paper, Fully Convolutional Networks for Semantic Segmentation, I found that they use an up-sampling layer to classify each pixel in to a class. I have two questions:

How do you understand the mathematics behind the de-convolution operation?
Why do we use an upsampling layer? Is it for extract more global context?


texts: How do I assess whether two time series change together?

texts:
In this example, at timepoint 5, both signals move up together. I would like to quantify these similar movements, and ideally disregard the parts where the signals are almost constant.
What correlation or similarity measures would be best here?


texts:
I would like to make a join between on 3 columns and also on the rows without using pandas.
Anyone have an idea?
listOfToJoin= ['ColA','ColB','ColC']

columns = ['ColA','ColB','ColC','ColD','ColE']
Rows = [[234, 343, 49, 54609, 499],
        [23, 56, 765, 54609, 432]]

the result should be :
final_Columns = ['ColA_ColB_ColC','ColD','ColE']
final_Rows = [[234_343_49, 54609, 499],
              [23_56_765, 54609, 432]]
```


texts: Changing order of input dimension in Tensorflow 3-D Layers

texts:
According to the official documentation of tf.keras.layers.Conv3D

5+D tensor with shape: batch_shape + (channels, conv_dim1, conv_dim2,
conv_dim3) if data_format='channels_first' or 5+D tensor with shape:
batch_shape + (conv_dim1, conv_dim2, conv_dim3, channels) if
data_format='channels_last'

.
Now the whole idea around channels and batch shape makes sense, but will changing the general order of
(conv_dim1, conv_dim2,conv_dim2) as (x,y,z) to say (z,x,y) affect the performance.
Does Conv3D worry about order of x-y-z dimension ?
I was training a U-net segmentation model and upon changing the order of axis I saw difference in performance. (x,y,z) order converges faster as compared to (y,x,z).
I just wanted to make sure what's the correct way..


texts:
I'm trying to solve a 'negation-like' classification problem, where I need to classify whether a certain word within the context has negative or positive label.

For example, how to identify whether a keyword has been prescribed or only mentioned:

['Aspirin has been prescribed to a patients'] -> {[('key_word': 'aspirin', 'prescribed': TRUE)]}

['If symptoms continue, the patient should consider taking Omeprazol'] -> {[('key_word': 'omeprazol', 'prescribed': FALSE)]}

[‘The plan is for him to commence 25mg of Trazodone as soon as he gets better.’] -> {[('key_word': 'trazodone ', 'prescribed': FALSE)]}

['her current meds are: sertraline 200 mg and olanzapine 5 mg'] -> {[('key_word': 'sertraline', 'prescribed': TRUE), ('key_word': 'olanzapine', 'prescribed': TRUE)]}

['if she continues to be depressed, then she needs to be started on Risperidone'] -> {[('key_word': 'risperidone', 'prescribed': FALSE)]}

I have a training data set for this task, but it is not clear how to formulate the classification problem. It is similar to sentiment classification problem, but here instead of predicting a label of the entire sentence, I need to predict a label of a certain keyword based on the context.

Any ideas?


texts:
I am currently reviewing some concepts related to Machine Learning, and I started to wonder about the hyperparameter selection of K-NN classifier.
Suppose you need to solve a classification task with a number of classes equal to M: I was thinking that the best choice for the parameter K of our classifier should be for K > M.
In this way, we are avoiding all the pathological cases in which a sample may be in the middle of all the M classes and then have a tie. For instance, consider the following example in which we have M=3 and each geometrical shape represents a class:

Assume that K<=M: for sure you will have a tie for a sample in the middle of samples 1, 2 and 3. This tie could be avoided if K > M.
Clearly this is just a toy example, but I think it is sufficient to illustrate my thoughts. I have tried to look for an answer but I wasn't able to find any resource mentioning this, am I wrong in some way or this reasoning may be sound?


texts: Creating new columns by iterating over rows in pandas dataframe

texts:
I have a pandas data frame (X11) like this: In actual I have 99 columns up to dx99

    dx1      dx2    dx3    dx4
0   25041   40391   5856    0
1   25041   40391   25081   5856
2   25041   40391   42822   0
3   25061   40391   0       0
4   25041   40391   0       5856
5   40391   25002   5856    3569

I want to create additional column(s) for cell values like 25041,40391,5856 etc. So there will be a column 25041 with value as 1 or 0 if 25041 occurs in that particular row in any dxs columns. I am using this code and it works when number of rows are less.

mat = X11.as_matrix(columns=None)
values, counts = np.unique(mat.astype(str), return_counts=True)

for x in values:
    X11[x] = X11.isin([x]).any(1).astype(int)

I am getting result like this:

dx1     dx2     dx3    dx4  0   25002   25041   25061   25081   3569    40391   42822   5856
25041   40391   5856    0   0   0       1       0       0       0          1        0       1
25041   40391   25081  5856 0   0       1       0       1       0            1      0       1
25041   40391   42822   0   0   0       1       0       0       0           1       1       0
25061   40391   0       0   0   0       0       1       0       0          1        0       0
25041   40391   0    5856   0   0       1       0       0       0          1        0       1
40391   25002 5856   3569   0   1       0       0       0       1          1        0       1

When number of rows are many thousands or in millions, it hangs and takes forever and I am not getting any result. Please see that cell values are not unique to column, instead repeating in multi columns. For ex, 40391 is occurring in dx1 as well as in dx2 and so on for 0 and 5856 etc. Any idea how to improve the logic mentioned above?


texts:
I have a feature for machine learning (using methods like SVM, naive bayes, neural network and random forest) called member duration as follows:
Should I make it as numerical or categorical data?


texts:
I have been coding my own multi layer perceptron in MATLAB and it can be compiled without error. My training data features,x, has values from 1 to 360, and training data output, y, has the value of sin(x).

The thing is my MLP only decreases the cost for the first few iterations and will get stuck at 0.5. I have tried including momentum but it doesn't help and increasing the layers or increasing the neurons do not help at all. I am not sure why this is happening ....

I have uploaded the files for your reference here.

The summary of my code is
1) I normalize my input data either using min-max or zscore

2) Initialize random weights and bias within the range of -1 to 1

for i = 1:length(nodesateachlayer)-1
    weights{i} = 2*rand(nodesateachlayer(i),nodesateachlayer(i+1))-1;
    bias{i} = 2*rand(nodesateachlayer(i+1),1)-1;
end

3) then I do a forward pass where input is multiplied by weights and added with bias then activated by a transfer function (sigmoid)

for i = 2:length(nodesateachlayer)
        stored{i} = nactivate(bsxfun(@plus,(weights{i-1}'*stored{i-1}),bias{i-1}),activation);
    end

4) then calculate the error then do a backward pass

dedp = 1/length(normy)*error;
    for i = length(stored)-1:-1:1
        dpds = derivative(stored{i+1},activation);
        deds = dpds'.*dedp;
        dedw = stored{i}*deds;
        dedb = ones(1,rowno)*deds;
        dedp = (weights{i}*deds')';
        weights{i}=weights{i}-rate.*dedw;
        bias{i}=bsxfun(@minus,bias{i},rate.*dedb');
    end

5) I have the cost plotted out at every iteration to see the descent

I assume there is something wrong with the code so where could the error possibly lies in


texts: Why is data science not yet widely applied to Law?

texts: Law (judiciary) contains such a huge corpus to apply NLP to, but yet there are only search engines designed for Law. Why is NLP not yet extensively applied? Is it because of dimensionality?


texts:
Nothing to do with number recognition in the classical 'hand-written' sense

Disclaimer above to avoid this being counted as a repeat.

I have a selection of 96 serial numbers, and a separate selection of >220 serial numbers. Within the larger set typically resides the smaller set (not always though), but also ~ 120 incorrect numbers.

See below for an example - for the record I have matched things up as best as I can... the correct number is first, the 'possibles' are in parenthesis at right:

21490 (21490, 21400, 21498, 21499, 21480, 21488)

21491 (21401, 21481, 1401)

21492 (21492, 21402)

This set gives a good example of the type of thing i'm seeing:

Number being misread the same way (0--> 9 and 8)
Sometimes a number is being missed entirely
Sometimes the right number isn't read at all...

It's not limited to 0, 8s and 9s, but these are the worst, so I'd like to try and understand which numeric characters are problematic (give them all a score), and build a model which takes a number, and knows a list of numbers it CAN be, and give me what number it should be, ideally with a confidence metric.

Anyone done this before and have any ideas?


texts:
Does this exist? What algorithm or combinations of algorithms would be able to classify images without supervision?

For example if you have many pictures of cats and dogs, then without being trained to distinguish them first, it would go through the images and realize there are two distinctive groups of images (either cat or dog)

Or if you have many random pictures but a good portion of them have trees, then going through the images, the algorithm realizes there is one consistent pattern within some of the images (trees)


texts: What is the easiest way to duplicate my model and run in 10 cloud machines?

texts:
I have built a model in a cloud machine(google cloud). It runs for a few hours to a day. I need to scan some parameters like learning rate and batch size.

How do I duplicate my compute engine, run the model with different parameters, collect the results and turn them off?

Edit:
I have a neural network model, it runs for 24 hours. Usually without cloud setting I would do a grid search: learning rate in {0.001, 0.003, 0.1} and batch size in {32,64,128}. This would take 9 days.

With cloud computing I can do this grid search in 24 hours. I need to manually do the followings. Save my original compute engine into a snapshot and create 8 compute engine from the snapshot, start all engines. Run each model with a different parameter. Copy the result to the original compute engine. Close after copy.

The question is how do I automate this? Answers in gcp or aws are welcomed.


texts: illumination problem for face landmark detection

texts: I trained dlib face landmark detection model using ibug and yaleB datasets. Accuracy of the model is dropping when a bright light or a shadow appears on the side face. Used default parameters and similar images were included while training. Is there any solution to overcome the problem.


texts: Is classifier able to say there's no-such-case?

texts:
I am a starter in ML and I need some help...
The problem
Assume that I have a classifier which can classify left hand / right hand well.
I am curious whether it can decide whether there's a hand in the image?
Because if it is not the case, I have to collect additional data that are labeled as "no-hand". But there's so many things that are "no-hand" such like dogs, cats, flies... I don't think it is possible to collect all those possible "no-hand" data..
My intuition
Would it be reasonable that we use some activation functions like [sigmoid] as the final outputs for the classifier(neural network) and we consider the outputs for both left/right classes as the scores that classifier think how far the input is from the left/right hands classes.
Then if the outputs are below some threshold for both left hand and right hand we can say there's no hand in this image input.


texts: Which Big-Data Frameworks have most simple interfaces?

texts:
I found that Apache-Spark has pretty much simple interface and easy to use. But I want to know about other interfaces.

Can anyone give me a ranking of Big-Data frameworks in base of simplicity of their interfaces. also this is useful to express most simple and complex interfaces in base of your experiences.

Definitely this question is about some frameworks with same tasks. For example a selection between Flink and Spark just in your opinion. Detailed comparison is so lengthy and this is not my purpose. Just a selection or ranking on your opinions is sufficient.

Thanks


texts: Handling categorical data with more over 100 unique classes

texts:
I am working with a pure categorical data set. And some classes have more than 100 unique values.
I could not find any appropriate encoding possibility. So I created a SQL table, where each value got its ID. Then I extracted the IDs and used it in ML Classification.
However the results are poor.
So anyone has an idea how to encode such values better?


texts: An alternative algorithm to sklearn KMeans that separates values by similarity?

texts:
I have the following dataset:

        node        bc cluster
1    russian  0.457039       1
48       man  0.286875       1
155    woman  0.129939       0
3        bit  0.092721       0
5      write  0.065424       0
98       age  0.064347       0
97     escap  0.062675       0
74      game  0.062606       0

Then I perform kMeans clustering by bc value to separate the nodes into two different groups. Right now with the code below I get the result above (the clustering result is in the cluster column).

    bc_df = pd.DataFrame({"node": bc_nodes, "bc": bc_values})
    bc_df = bc_df.sort_values("bc", ascending=False)
    km = KMeans(n_clusters=2).fit(bc_df[['bc']])
    bc_df.loc[:,'cluster'] = km.labels_
    print(bc_df.head(8))

Which is pretty good, but I would like it to work slightly differently and to select the first 4 nodes into the first cluster and then the other ones in the 2nd one, because they are more similar to each other.

Can I do some adjustment to kMeans or maybe you know another algorithm in sklearn that can do that?


texts: Is this a Q-learning algorithm or just brute force?

texts:
I have been playing with an algorithm that learns how to play tictactoe. The basic pseudocode is:

repeat many thousand times {
  repeat until game is over {
    if(board layout is unknown or exploring) {
      move randomly
    } else {
      move in location which historically gives highest reward
    }
  }

  for each step in the game {
    determine board layout for current step
    if(board layout is unknown) {
      add board layout to memory
    }
    update reward for board layout based on game outcome
  }
}

now play a human and win :-)

Exploration: in the beginning the algorithm explores aggresively, and this reduces linearly. After say a thousand games it only explores in 10% of the moves. All other moves are based on exploitation of previous rewards.

Rewards: if the game resulted in a win, then award 10 points. If the game resulted in a draw, 0 points, otherwise -5 points.  Actually, these rewards can be "tuned", so that if the game was shorter and it was won, then award more points or if it was longer award less points.  This way the algorithm prefers winning quickly. That means that it learns to win as soon as possible, rather than aiming to win later on.  That is important so that it doesn't miss winning immediately - if it missed such a move the opponent would likely a) move there to avoid letting the AI win next time, and b) think the algorithm was stupid because it missed an "obvious" win.

This algorithm does indeed learn, so I can class it as a maching learning algorithm.

I think, but I am not sure, that it is a reinforced learning algorithm. However, according to https://www.cse.unsw.edu.au/~cs9417ml/RL1/tdlearning.html it is not temporal difference learning, because it doesn't estimate the rewards until the end, and it should be estimating the reward as it goes along. That might mean that it is not reinforced learning.

Question 1: Can I successfully argue that I am estimating the reward based on history, and still claim the algorithm is reinforced learning or even Q-learning?

Question 2: If I replace the reward lookup which is based on the board layout, with a neural network, where the board layout is the input and the reward is the output, could the algorithm be regarded as deep reinforcement learning?

Question 3: I'd don't think that I have either a learning rate or a discount factor. Is that important?

I noticed that the algorithm is pretty useless unless I train it with at least every move which an opponent tries. So in a way it feels like it's using brute force rather than really "learning". This makes me question whether or not machine learning tictactoe is really learning.  I agree that using a neural network to learn image recognition can be classed as learning because when it sees an unknown image it is able to state its classification. But that is quite useless for games like tictactoe where similar looking board layouts are totally unrelated (one may lead to a win, the other may lead to a loss). So...

Question 4: Can tictactoe algorithms be classed as real learning rather than simply brute force?

Update: regarding rewards... when the algorithm is deciding where to go, it works out the reward for each position as follows:

var total = winRewards + drawRewards + lossRewards;
move.reward = (100*(winRewards/total)) + (10*(drawRewards/total)) + (-1*(lossRewards/total));

I divide by the total number of points (for each move), because otherwise it seems to learn that one place is GREAT and doesn't give other ones a chance.  This way, we work out the win ratio regardless of how often it's been played. It's normalised in comparison to the others.

The code is available here: https://github.com/maxant/tictactoe/blob/master/ai.js

UPDATE #2: I have since figured out that this algorithm cannot be classed as using brute force because it doesn't actually learn that many games before becoming an expert. Details here: http://blog.maxant.co.uk/pebble/2018/04/11/1523468336936.html