texts: What is the function of the hidden layer in a neural network mapping to?

texts:
The hidden layers of the neural network which are in between the input layer and the output layer take in input data and apply a function to churn out data from each node which is then weighed by the next layer.

The way I've seen it explained is for example to predict whether a image is a bus , the nodes learn whether there is a wheel , a tire etc or to predict if a person likes a document, the hidden layer nodes map to whether the topic is sports, history etc.

However the "function" that the hidden layer nodes apply, what exactly are they mapping to i.e what is the dependent variable? Is it the output variable that the model is fed ? For example if I'm trying to predict if the image given is that of a dog or cat , then does the hidden layer map  the input data to the output variable(dog or cat) ? How exactly does the hidden layer learn useful features? How do the nodes learn different features rather than the same one?


texts:
I already asked this question in stack overflow, but got response from experts, to post this question here, please help me to understand this concept...
I am trying to understand RPN network in Faster RCNN.
I understand the concept of RPN network,

Pass the input images to the pre trained CNN, and get the output as feature maps
Make fixed size of the feature maps
Extract anchors (3 different scales and ratio for every sliding window) from the fixed size feature maps.
Use two 1×1 Fully connected NN to find the background or object and the bounding box coordinates (4 values)
Calculate IOU for Anchors bounding box with Ground Truth bounding box, if IOU>0.7, then the anchor has object, otherwise, the anchor has background.

The theme for RPN is to give the region proposals which have objects.
But, I do not understand the input and the output structure.
For example, I have 50 images, each images having 5 to 6 objects, and labeling informations(coordinates of each objects).
How do I generate target values, to train PRN Network...
In all the blogs, they shows the architecture as feed the entire image to the pre trained CNN.
And, the output of RPN, the model has to tell whether the anchor has object or not, and also predict the bounding box for the object in the anchor.
For this, how to prepare the input and target/output values like we do in dog/cat or dog/cat/car classification problem.
Let me correct if I am not correct,
Is that, we have to crop all the objects in every image and do binary classification as object vs background for classifying the anchor has object or not
And, Is that, we have to give the ground truth value as target for every cropped objects from all images in the dataset, so that the RPN network trained well to predict the bounding box for the object in every anchor.
Hope, I clearly explained my doubts.
Help me to learn this concept, Thank you


texts: Is there publicly available dataset containing real world people names? I’m looking for dataset containing names from different regions and cultures. I want to use it for research on name spellchecking, finding spelling similarity without bias towards particular language.


texts: Divergence of Specificity and Sensitivity

texts:
I am working on a ML classification project in healthcare.
The data is imbalanced, and I decided to start modeling with tree-based algorithms such as (Balanced)RandomForest and XGBoost.
While checking the model's performance and the effect of model parameters on the outcome, I observed a strange phenomenon: there is a considerable gap between model's sensitivity and specificity;
the more one increases the more the other decreases. For example, the following picture shows the variation of specificity and sensitivity as the 'max_leaf_nodes' parameter in RF model changes.

Is such a divergence abnormal, or it could actually be due to the data and quality of the features?
What one can do to overcome this problem?


texts: How to cluster a set of objects each with its' own set of data?

texts:
I have a problem where I have a set of objects (say smartphones for ease of explanation). For each phone, we collected several characteristic features (such as duration of calls, how many calls made, received etc) over a period of 3 months (say). Now based on these collected datasets, we need to group the phones into clusters. The situation is kind of like in the following figure:

My first thought was to follow what we do during working with images. Make a vector out of the image matrix and then do the clustering. In my case, we make the vector by maintaining the temporal sequence. So for each phone, if the size of the dataset be m x p and there are n such phones, we build a dataset with size n x (m x p). However, I cannot decide if there is any problem with this approach or whether it will achieve what we want.
I am looking for some ideas.Any suggestions would be greatly appreciated.
Thanks and regards.


texts: How to select best feature set and not ranking for tree based models?

texts:
I am currently using feature selection approaches like filter, wrapper, embedded etc.

All these methods give different set of features and I rank them based on their frequency of occurrence in other feature selection approach.

Ex: If Age occurs in all 6 feature selection algorithm, then they will have a rank as 6. If gender occurs only in 2 feature selection algorithm, its rank would be 2.

So I arrange them in descending order and choose the features which has occurred in at least 3-4 feature selection algorithm.

But where I am trying to seek your help is

1) Is there any systematic way where I get only a subset of features which returns best output? I thought genetic algorithm for feature selection will return a feature subset which will give high output. But unfortunately it lists all features with their importance. I don't wish to define a threshold myself to select few. Example is currently Xgboost produces an F1-score of 81 with around 27-28 features but when I manually play around, it gives around 81.9 or 82.1 with just 6-7 features.  So I would like to have a way to select this 6-7 features automatically/systematically without human intervention

2) Is there any algorithm, like RFE but which does exhaustive search like genetic algorithm and finally returns us the best feature set? Mix of RFE and genetic algorithm is what I need

3) I wish to have an exhaustive search in feature space and finally provide me the best feature set which will provide best f1-SCORE which is my objective.

I am looking for a systematic way/algorithmic way to do this rather than me deciding to pick features which occurs at least in 3 feat selection algorithms

Hope my question is clear. Can you help me with this?


texts: Could data from a test set 'leak' into predictor during PCA?

texts:
After reading this article I have got a question about PCA.

Author was talking about whether to use test set while computing PCA.

  But, few important points to understand:

  1) We should not combine the train and test set to obtain PCA components
  of whole data at once. Because, this would violate the entire
  assumption of generalization since test data would get ‘leaked’ into
  the training set. In other words, the test data set would no longer
  remain ‘unseen’. Eventually, this will hammer down the generalization
  capability of the model.

  2) We should not perform PCA on test and train
  data sets separately. Because, the resultant vectors from train and
  test PCAs will have different directions ( due to unequal variance).
  Due to this, we’ll end up comparing data registered on different axes.
  Therefore, the resulting vectors from train and test data should have
  same axes.

Author mentioned "Because, this would violate the entire assumption of generalization since test data would get ‘leaked’ into the training set. In other words, the test data set would no longer remain ‘unseen’."

As I understand it... test data can only affect which PC(s) we will choose ( because of changed variance) but data can't leak really( let's say only metadata leak). For each observation we still have data only from it's own previous variables(predictors) just in lower dimensionality(projected). Am I right? or during calculations of matrixes data from test set can leak into?


texts: I have done a 10 fold Cross Validation on my data and have selected the best model from the results. With cross validation, I will have 10 models trained from different folds of the data. For the final model to use, should I take the average of the models or just fit a model for the entire training set?


texts: Removing duplicates and keeping the last entry in pandas

texts:
I have a table and I'm trying to remove all the duplicate and keep the
the rows that has the latest datestamp.

Here is the table:

email address       orgin_date   new_opt_in_date   datestamp
123@ax.tu            1/1/1900     1/1/1900          3/15/2016
123@ax.tu            1/1/1900     1/1/1900          3/15/2016
iron_man@metrix.com  2/15/2015    3/5/2015          6/6/2017
iron_man@metrix.com  2/15/2015    3/5/2015          7/6/2018
sleep@dort.st        2/15/2015    3/5/201           7/6/2018
sleep@dort.st        2/15/2015    3/5/201           5/6/2018

I'm trying to keep only the data that has the recent datestamp and the
output will like this:

 email address       orgin_date   new_opt_in_date   datestamp
 123@ax.tu            1/1/1900     1/1/1900          3/15/2016
 iron_man@metrix.com  2/15/2015    3/5/2015          6/6/2017
 sleep@dort.st        2/15/2015    3/5/201           7/6/2018

I use this formula:

df.drop_duplicates(keep = False)

or this one:

df1 = df.drop_duplicates(subset
['emailaddress', 'orgin_date', 'new_opt_in_date','datestamp'],keep='first')
print(df1)

but nothing works


texts:
I have time series data stored in a data frame as follows:

Time, c1, c2, c3

0, 0.55, 0.4 , 0.3

1, 0.8 , 0.1 , 0.6

2,  0.9 , 0.5 , 0.7

....

And I want to predict the values of c1, c2, c3 at the next time interval. How can I implement a python code to predict values using SVR (Given above structure)?


texts: How exactly does DropOut work with convolutional layers?

texts:
Dropout (paper, explanation) sets the output of some neurons to zero. So for a MLP, you could have the following architecture for the Iris flower dataset:

4 : 50 (tanh) : dropout (0.5) : 20 (tanh) : 3 (softmax)

It would work like this:

$$softmax(W_3 \cdot \tanh(W_2 \cdot \text{mask}(D, \tanh(W_1 \cdot input\_vector)))$$

with $input\_vector \in \mathbb{R}^{4 \times 1}$, $W_1 \in \mathbb{R}^{50 \times 4}$, $D \in \{0, 1\}^{50 \times 1}$, $W_2 \in \mathbb{R}^{20 \times 50}$, $W_3 \in \mathbb{R}^{20 \times 3}$ (ignoring biases for the sake of simplictiy).

With $D = (d)_{ij}$ and

$$d_{ij} \sim B(1, p=0.5)$$

where the $\text{mask}(D, M)$ operation multiplies $D$ point-wise with $M$ (see Hadamard product).

Hence we just sample the matrix $D$ each time and thus the dropout becomes a multiplication of a node with 0.

But for CNNs, it is not clear to me what exactly is dropped out. I can see three possibilities:

Dropping complete feature maps (hence a kernel)
Dropping one element of a kernel (replacing an element of a kernel by 0)
Dropping one element of a feature map

Please add a reference / quote to your answer.

My thoughts

I think Lasagne does (3) (see code). This might be the simplest to implement. However, closer to the original idea might be (1).

Seems to be similar for Caffe (see code). For tensorflow, the user has to decide (code - I'm not sure what happens when noise_shape=None is passed).

How it should be

(2) and (3) don't make much sense as it would cause the network to add invariance to spacial positions, which is probably not desired. Hence (1) is the only variant which makes sense. But I'm not sure what happens if you use the default implementation.


texts: Methods for analyzing multiple time series

texts:
My agenda is finding patterns and possible a model that describe my data.
My data is comprised of multiple time-series uni-variate sets.

data_set = [pd.Series(np.random.normal(size=10), range(10)),
            pd.Series(np.random.normal(size=10), range(10)),
            pd.Series(np.random.normal(size=10), range(10)),...]

Each of these series follows some time-series assumptions - meaning there is a dependency between X and X-1, X-2 .. at each series.
All of the series represent similar process, so they have some-what similar properties(value scale, change points, seasonality) but they are not identical.

As to my understanding all time-series models are good for learning one time-series with no implication on other series.
Is there a way for fitting a model that is best for all series?
What is the method for learning/model across multiple related time series?


texts: What is the difference between a regular Linear Regression model and xgboost with objective set to "reg:linear"?

texts: As I understand it, a regular linear regression model already minimizes for squared error, which means that it is the theoretical best prediction for this metric.  Does xgboost's "reg:linear" objective do something other than minimize squared error?


texts:
I'm mainly interested in creative things, but also interested in the science behind viewership and marketing a piece of media. Think about having a stable career in data science, where do you think I should begin? Are media analytics as big as others like business intelligence? Ive checked online courses, like google analytics academy and others like these:

https://www.edx.org/xseries/data-science-analytics-context

https://www.coursera.org/specializations/jhudatascience

Is this the right path to take? And is it easy to find jobs in such a field even if the person is a beginner in programming?


texts:
I have a multi-class prediction problem
but the 300classes is imbalanced
should I make it balance all 300 class will predict the better result?
is there an easier method to do this job?
if I'm using the random-forest imbalance dataset is matter?


texts: How to detect if a person is interacting (as in touching) some object in an image?

texts:
I am currently working on a human interaction problem, which tries to identify if a person touched a predefined object.

Current Approach: I am using open-pose to estimate the pose of the person then manipulating the keypoints returned into some logic (simple if else statements) which tells whether person interacted or not. However it seems to be a dirty workaround to me and is not a full proof solution. So is there any better approach I can go for?

Is there any predefined model or white paper I can refer for this problem?

I am struggling on how to detect the pose of a person in an image? Is there a way to classify the pose of a person in three categories like front, back and side?


texts: What are some method for pre-processing data in OCR?

texts:
I have a dataset for a supervised learning task.

Each row is a vector with a value of pixmap value in a range [0,255] of gray colormap, each vector is labeled with a character. I have to assign each vector with a character.

My Question:

What are some methods that I can try to pre-process the data to gain better accuracy?


texts: Keras custom loss - operation on additional data

texts:
I am trying to create a custom loss function for a Keras regression task.

I am predicting the points scored per minute in a game, and training on "matches" of variable lengths, in minutes. In an attempt to help the model learn, I would like to include the number of minutes the group played as part of the loss function, so we can properly "punish" for missing on groups that play for longer durations.

I know I need to use Keras Backend/tensor operations, but I'm stuck on how to prepare the minutes/how to multiply the minutes tensor. Here's some basic code what I have so far (pseudo code/skipped where not important):

def penalized_loss(minutes):

    # minutes is np.array - do we need to convert to tensor?
    #    Is this useful/how to do it?
    minutes_tensor = K.constants(minutes)

    def loss(y_true, y_pred):

        # Normal mean_squared_error - this makes sense to me
        lost = K.mean(K.square(y_pred - y_true), axis=-1)

        # Would like something like this where minutes are included, but as a valid tensor operation
        not_valid_result = K.mean(K.square(y_pred - y_true) * minutes, axis=-1)

        # Is this closer to what we need to do?
        maybe_closer_result = K.mean(K.dot(K.square(y_pred - y_true), minutes),  axis=-1)

        return lost

    return loss

# minutes is np.array, shape (12024,)
# input/predictions also have shape 12024

def baseline_model():
    model = Sequential()
    # Create the model....

    # Compile model
    model.compile(loss=[penalized_loss(minutes)], optimizer='adam')
    return model

The closure to pass in minutes works as I expect, I'm just stuck on how to do the * minutes portion of the loss calculation. I've looked through the docs and I feel like I'm missing something easy/obvious. Can anyone point me in the right direction?


texts: How do PoseNet identifies the key points of human pose estimation?

texts: PoseNet is a deep learning TensorFlow model that allows you to estimate and track human poses by detecting body parts such as elbows, hips, wrists, knees, and ankles , It uses either MobileNet or ResNet architectures to find the keypoints . I want brief that how it identifies the keypoints, what is inside which is detecting it ?


texts:
with regards to the Logistic Regression cost function of:

And hypothesis:

Is there a way to tell the +/- of the error for how "confident" the hypothesis is?

E.g. if the +/- of the error was 0.1, I would know that if my hypothesis predicted 0.4 it could be 0.1 greater (0.5) or 0.1 less (0.3)

This is for binary classification


texts: Predicting a random variable Y will increase within n minutes if random variable X increases

texts:
I am starting to learn machine learning and I would like advice of how to model that random measure Y will increase within n minutes if random measure X increases.

X and Y have different units (GBP and USD) so I am using returns (percentage change) to make them the same unit.

My data looks like this

Time, X, Y
00:00, 1%, 0%
00:01, 0%, 1%
00:02, 2%, 0%
00:03, 0%, 2%

Just from observing this data you can see that Y increases after one second after X increases but not sure the best way how to model this!


texts: I have medical data set with different types of attributes and I have to do the classification based on attributes: there are no class labels in this data set.  Suggest to me which algorithm works better for this dataset.


texts: Detect if word is «common English» word or slang word

texts:
I have a huge list of short phrases, for example:

sql server data analysis # SQL is not a common word
bodybuilding # common word
export opml # opml is not a common word
best ocr mac # ocr and mac are not common words

I want to detect if word is not a common word and should not be processes further.

I've tried to do this with NLTK, but it gives strange results:

result = word in nltk.corpus.words.words()

sql = false
iso = true
mac = true

Is there a better way to do this?


texts: Alternatives to TF-IDF and Cosine Similarity when comparing documents of differing formats

texts:
I've been working on a small, personal project which takes a user's job skills and suggests the most ideal career for them based on those skills. I use a database of job listings to achieve this. At the moment, the code works as follows:

1) Process the text of each job listing to extract skills that are mentioned in the listing

2) For each career (e.g. "Data Analyst"), combine the processed text of the job listings for that career into one document

3) Calculate the TF-IDF of each skill within the career documents

After this, I'm not sure which method I should use to rank careers based on a list of a user's skills. The most popular method that I've seen would be to treat the user's skills as a document as well, then to calculate the TF-IDF for the skill document, and use something like cosine similarity to calculate the similarity between the skill document and each career document.

This doesn't seem like the ideal solution to me, since cosine similarity is best used when comparing two documents of the same format. For that matter, TF-IDF doesn't seem like the appropriate metric to apply to the user's skill list at all. For instance, if a user adds additional skills to their list, the TF for each skill will drop. In reality, I don't care what the frequency of the skills are in the user's skills list -- I just care that they have those skills (and maybe how well they know those skills).

It seems like a better metric would be to do the following:

1) For each skill that the user has, calculate the TF-IDF of that skill in the career documents

2) For each career, sum the TF-IDF results for all of the user's skill

3) Rank career based on the above sum

Am I thinking along the right lines here? If so, are there any algorithms that work along these lines, but are more sophisticated than a simple sum? Thanks for the help!


texts: Hindsight Experience Replay: what the reward w.r.t. to sample goal means

texts:
Referring to the paper on Hindsight Experience Replay

Is it right that sampled goals which are visited states should be followed by a positive (or non-negative) rewards in order to allow an agent learn?

On page 5 of the paper, a "Algorithm 1 Hindsight Experience Replay (HER)" scheme reads in particular:

for t = 0, T-1 do
    r_t := r(s_t, a_t, g)
    Store the transition (s_t || g, a_t, r_t, s_(t+1) || g) in R

    Sample a set of additional goals for replay G := S(current episode)
    for g' ∈ G do
        r' := r(s_t, a_t, g')
        Store the transition (St||g', a_t, r', s_(t+1)||g') in R
    end for
end for

where:

  g  : current goal
  R  : replay buffer

  All other symbols with a dash indicate that they were sampled in addition to the actual current goal within the current episode.

It means (as long as I understand) that for the sampled goals (g') the reward is now a function of action taken in state given the sampled goal. It is not very clear whether the agent will learn the task in case the reward is still the old function (which is non-positive for all states that are different from the final goal).

As an example, in a grid-world an agent gets -1 reward while not in the final cell of its destination, but with the new goals introduced, the agent's reward with respect to its current state is not r, but r' (reward after reaching goal).

Illustrating example (grid-world):


texts:
I use a brute-force mechanism to determine optimal hidden layers/neurons by incrementing the layers/neurons by 1 up to some maximums and then picking the optimal counts from the best performing model.  My question here is about the starting point for the hidden neuron count of this "brute-force" process.  What are some good mechanisms for determining what neuron count to start at such that I am not likely to miss some minimum optimal count?

For example:

Say I have 8 inputs that are NOT linearly related.  If I take (number of inputs: 8) + 1 (as some post suggest) for my starting point of incrementing the number of neurons within the layers, would I likely be missing the optimal neurons count as it may be less than 9 neurons?

The brute-force mechanism I am using sets a max number of neurons per hidden layer, so if I have 8 inputs, and I started with a range of 9 to 30 neurons per hidden layer, I would have 9 neurons in the first hidden layer which would increment to 30 and then roll back to 9 when starting to test with a second hidden layer:

Cycle 1
Hidden Layers = 1
Hidden Neurons = 9
...
Hidden Layers = 1
Hidden Neurons = 30

Cycle 2
Hidden Layers = 2
Hidden Neurons = 9
...
Hidden Layers = 2
Hidden Neurons = 30

Cycle 3
Hidden Layers = 3
Hidden Neurons = 9
...
Hidden Layers = 3
Hidden Neurons = 30

I admit this has inefficiencies as each layer may not need the same number of neurons, but the point of this post is to try and improve this mechanism by minimizing the number of layers/neurons combinations that need to be tested to determine the "optimal" counts.

EDIT:

So just to clarify the question is about determining what a good minimum starting point would be if using the brute-force mechanism outlined above regardless of data domain (hence the input + 1 example).


texts: Why yolo4 pytorch re-training loss seems high as like first time training?

texts:
I had a setup a yolo4 pytorch framework in google colab by cloning git clone https://github.com/roboflow-ai/pytorch-YOLOv4.git.
I generated checkpoints by giving training. As we need more robust training model, I given training again with assigning pretrained checkpoints but it seems loss started with high value as like first time training.
Code is for training !python train.py -b 2 -s 1 -l 0.001 -g 0 -pretrained ./Yolov4_epoch100_latest.pth -classes 1 -dir ./train -epochs 100.
Not sure if my pretrained checkpoint is used in 2end training? If it is used then why second time in starting training loss value seems high like first time training? Please share your thought if you have any


texts: Training NLP with multiple text input features

texts:
Question:

How can I train a NLP model with discrete labels that is based on multiple text input features?

Background:

I'm trying to predict the difficulty of a 4-option multiple choice exam question (probability of a test-taker selecting the correct response) based on the text of the question along with its possible responses. I'm hoping to be able to take into account how some incorrect yet convincing responses, the exact subject of which is relative to the content of the question, can skew the difficulty of the exam question, as well as how the wording of the question can make the question misleading.

Intuition, Attempts and Options:

My intuition is that the content of the question and its responses are both significant in the prediction of its difficulty. However, when using a library like Spacy, NLTK or Textacy, training seems to be done on only one text column at a time. I'm looking at potentially five text columns at a time, or two if I concatenate the question responses together.

I haven't been able to find much on the topic, but here is an attempt I found. I thought this attempt was flawed because they were just doing a single-column train multiple times, and for example training the City against a salary value and concatenating that to your train of Job Description against salary value is not going to give a meaningful improvement to your first model, since the Job Description did not depend on the City when training.

My options that I've found are:

Following the above attempt after all (which I think is flawed)
Concatenating my text features (which I can't understand why that would make sense in this case but seems to be the norm)
Eliminating some of my features entirely, such as narrowing the question down by subject matter and disposing of the question content and just training on the response options concatenated together (which also removes some very important information in the question content that can lead to prediction values)

Thoughts and advice? Is there a library that can make this easier? Thanks!


texts: Criteria used to create and select leaf nodes in sklearn

texts:
I just want to know the details of what (and how) is the criteria used by sklearn.tree.DecisionTreeClassifier to create leaf nodes. I know that the parameters criterion{“gini”, “entropy”}, default=”gini” and splitter{“best”, “random”}, default=”best” are used to split nodes. However, I could not find more information about the threshold used for spliting.
There are some methods involved in the creation of leaf nodes: post-pruning (cutting back the tree after a tree has been built) and pre-pruning (preventing overfitting by trying and stopping the tree-building process early).  It would be very useful to know more details about the criteria used for splitting to have a better understanding and be able to customize these models even more.


texts: How to filter out numbers from text documents

texts: I have several text files that have numbers in them.  It would be very difficult to filter out all the numbers by creating a Stop Word txt file with every number from 1 to 2010, plus very inefficient use of resources.  Is there a way in Orange to parse out all numbers from text documents?


texts: What tools to manage multiple machine learning projects in python do you use?

texts:
I will soon deal with multiple projects in python. Some of them have to run regularly (many times a day), they can take some time (many days) and they use/produce some data coming from and going to other servers.

I would like to know what tools you use to handle this kind of processes, knowing that they run on python. These tools could focus on:

workflow management (what task should be executed when ?)
data synchronization (database updates/synchronization)
resources monitoring (space on disk, RAM and CPU ?)
versioning (when these input/output arrived, should I treat them, or is there a new version ?)
visualization/monitoring (easy estimate the state of the processes, be alerted if an error occurred)

As an example, luigi tackle some of these problems.

Share links to articles or to tutorials if you have some in mind. I think this kind of challenges are faced by many people starting machine learning projects for production purposes.


texts: Evaluate Keras model with scikit-learn metrics

texts:
How does Keras calculate accuracy, precision, recall, and AUC?
I've created a model for categorical classification (i.e., multiple classes) by using keras.losses.CategoricalCrossentropy() as loss function, and in the model.compile() method I've specified the following metrics:

METRICS = [
    keras.metrics.CategoricalAccuracy(name='acc'),
    keras.metrics.Precision(name='precision'),
    keras.metrics.Recall(name='recall'),
    keras.metrics.AUC(name='auc'),
]

When I evaluate the model with model.evaluate(X_test, y_test, verbose=2) the result is the following:

  147/147 - 1s - loss: 0.5127 - acc: 0.8073 - precision: 0.8437 - recall: 0.7696 - auc: 0.9537

Then, I want to compare its performance with the results obtained with other algorithms (e.g., DecisionTreeClassifier() implemented in scikit-learn).

To evaluate the other classifiers, I'm using the following:

from sklearn.metrics import balanced_accuracy_score
from sklearn.metrics import roc_auc_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score

print("AUC: %2.3f" % roc_auc_score(y_test, pred_bin))
print("ACC: %2.3f" % balanced_accuracy_score(lb.inverse_transform(y_test), pred))
print("PRE: %2.3f" % precision_score(lb.inverse_transform(y_test), pred, average='weighted'))
print("REC: %2.3f" % recall_score(lb.inverse_transform(y_test), pred, average='weighted'))

However, when I test the performance of my NN with this piece of code, I obtain the following results:

  ACC: 0.715 PRE: 0.801 REC: 0.807 AUC: 0.920

As you can note, the result differ from those obtained with model.evaluate() of Keras.
How can I fairly compare the different methods?


texts:
I have read many blog articles, research papers and watched many youtube videos, but it seems it is hard to find why experience replay is efficient.

I know that the experience replay stores (state, action, reward, newstate) and sample minibatch and update weights, however, what I don't understand is how come the bellman equation can be the target value and how the learning can be done with supervised learning the value of (reward + gamma * (maximum next action)) being target?

It would be great if you can suggest any articles or videos?
or explain it.


texts: Since Orange is just running python code under the hood... is it possible to set up and do my analysis in Orange but then export or output the underlying code for documentation or replication?


texts:
I have been working with DC-GAN to generate pairs of images based on WGAN paper. For which I referred fastai notebook on WGAN where the bias in the network has been set to False. Jeremy Howard in his video regarding the notebook asked the viewers to figure out the reason which remains unsolved for me.

Please explain why the bias has been set to False and when should bias be set to False.


texts: What kinds of statistical analyses and machine learning techniques are most useful for personal analytics projects?

texts:
I am just a beginner-level programmer in Python and MATLAB, and I have been looking at tutorials for Keras. Eventually, I want to make a project for my programming portfolio. Can you please share some ideas about how to get started with this project as well as any career advice about getting started with data science and machine learning?

Here's an example of a project idea I want to work towards: Machine learning applied to a series of daily logs and selfies to look for patterns of mood and activities that the user inputs as well as identify stress factors:

If implemented in a smartphone application, the program can be made to send notifications to the user when the algorithm determines it would be beneficial to the user's wellbeing. With the use of neural networks and other sources of data such as from Apple Health, more categories can be added to identify more specific moods such as anger, excitement, and sadness.


texts: Connect Orange 3.20 to postgresql database

texts: I installed orange 3.20 on windows 7. It works so far, the problem is connecting it to a server based Postgres database. While the connection can be made in the moment, when you try to load a table the message "missing extension quantile" comes up. A few problems are coming up with this message. It seems like it is not possible to install this extension on a windows server without a lot of stress. The extension seems not to be actual (version 2015) and is not compatible with the actual postgresql versions (which will cause trouble for future updates). Does anyone know if Orange will support this in the future and a convenient way to use Orange for windows 7?


texts:
The data I am working on has some really large price values and some really small values. What I did was first perform feature bagging on the data and got them labelled to (0,1) and then did Clustering on the data along with the labels found in the previous task.

Is this a right way to go?


texts: Compute an ROC for a hybrid model where only one of the model components computes class probabilities

texts:
I've created a hybrid model by taking an existing decision engine (TRUE/FALSE output) and supplementing it with a random forest classifier (TRUE/FALSE) model. The output of the hybrid model is produced by computing an OR from the predictions of the two models.

When evaluating the performance on historical data, I can produce a ROC for the random forest by computing the class probabilities based on the votes from all of the random trees combined. I cannot do this for the existing decision engine because it does not output probabilities. For the purpose of this analysis, I need to treat it as a black box.

How can I compute an ROC for the hybrid model? I would like to do this with R. I'm imagining that for each point on the ROC of the first model, I would need to compute the lift generated by adding the existing decision engine model.


texts: Use of cross validation for Polynomial Regression

texts:
I've two text files which contains my data.
One text file on X axis another text file on Y axis
Then using scatter function from python I did the data visualization
After that, I used polyfit function from python to get the curve which will fit my data
In that polyfit function we need to write degree of the polynomial we want eg. 2 or 3
Now let's consider I got 4 such a equations of degrees 2,3,4,5 respectively.
Now here comes my problem.
Among all those equations I got, how can I select the best equation which fits my data. I want to use cross validation here. Any high level library function can be use. My language is Python.


texts: How to use pca results for linear regression

texts:
I have a data set of 11 variables with allot of observations for each one. I want to make linear regression on the variables with the observed $\vec{y}=\alpha +\beta*\vec{X}$ when X is matrix. I'm trying to reduce my parameters so I activate pca algorithm on X. I get the "loading" data but i don't understand how to use it to get only four (for example) variables to estimate instead of 11.

somebody can help?


texts:
I am building an RNN by following Siraj Raval's video implementation. I've adapted it to use my dataset rather than importing one from a file. When the program gets to the loss function, it reports
    loss += -np.log(ps[t][y[t], 0])
    IndexError: index 7 is out of bounds for axis 0 with size 4

What does this mean and how can i fix this? Also, in Siraj's version, he calculates the loss as:

loss += -np.log(ps[t][y[t], 0])

Shouldn't it be this instead

loss += -np.log(ps[t]) * [y[t], 0]

as the cross entropy loss is L = -yln(yhat)?

My code:

import numpy as np

# Data Processing
x = np.array([
    # t/no. of inputs
    [1, 2, 3, 4, 5, 6],
    [7, 8, 9, 10, 11, 12],     # Samples
    [13, 14, 15, 16, 17, 18],
    [19, 20, 21, 22, 23, 24]]).T

# Model Parameters
numInputs = x.shape[1] # Yields 4
numNeurons = 8 # Yields 8
numEntries = x.shape[0] # Yields 6

u = np.random.random((numNeurons, numInputs))
v = np.random.random((numInputs, numNeurons))
w = np.random.random((numNeurons, numNeurons))
bh = np.zeros((numNeurons, 1))
bo = np.zeros((numInputs, 1))
atimeline = [] # Contains 6 timestep's woth of a's
yhattimeline = [] # Contains 6 timestep's worth of yhat's
hprev = np.zeros((8, 1)) # Contains previous a state

# Training
def loss(x, y, hprev):
    xs, hs, ys, ps = {}, {}, {}, {}
    hs[-1] = np.copy(hprev) # Copies hprev so hprev can still be used
                            # Adds key: -1, value: hprev, to the dict
    loss = 0
    xs[0] = np.zeros((numInputs, 1)) # Sets t = 0 to 0's
    for t in xrange(numEntries):
        xs[t + 1] = x[t] # Adds in data from x to the dict
        hs[t] = np.tanh(np.dot(u, xs[t]) + np.dot(w, hs[t - 1]) + bh)
        ys[t] = np.dot(v, hs[t]) + bo
        ps[t] = np.exp(ys[t]) / np.sum(np.exp(ys[t])) # Softmax
        loss += -np.log(ps[t][y[t], 0])
    du, dv, dw = np.zeros_like(u), np.zeros_like(v), np.zeros_like(w)
    dbh, dbo = np.zeros_like(bh), np.zeros_like(bo)
    dhnext = np.zeros_like(hs[0])
    for t in reversed(xrange(numEntries)):
        dy = np.copy(ps[t])
        dy[targets[t]] -= 1 # Derived from dL/dyhat
        dv += np.dot(dy, hs[t].T)
        dbo += dy
        dh = np.dot(v.T, dy) + dhnext
        dhraw = (1 - hs[t] * hs[t]) * dh # tanh
        dbh += dhraw
        du += np.dot(dhraw, xs[t].T)
        dw += np.dot(dhraw, xs[t - 1].T)
        dhnext = np.dot(w.T, dhraw)
        return du, dv, dw, dbo, dbh, hs[numEntries - 1]

u, v, w, dbo, dbh, hprev = loss(x, x, hprev)


texts: Extract locations from text - NER with Gazetteer, compare n-grams?

texts:
I have a (relatively) simple problem to solve, but don't know about the best practice how to solve it.
Given a list of several locations/places/point of interests
("Super Beer Bar", "London Stadium", "Madame Tussauds",...) and some random texts containing those places (i.e. "We've visited Madame Tussauds"), I like to check which places those random texts contain.
At first I thought this is an NER problem, but I would start with unsupervised learning here as I don't know much about those texts right now.
Characteristics of input texts: Relatively short (max 20 words), places might be misspelled sometimes.
My idea was to divide the texts into words, iterate over them and iterate over divided place names and check if there are matches, but this is not really efficient.
What would be the best approach to extract those places? And how would you deal with that problem later on, would to be training an NER later be the better approach?


texts: how to interpret predictions from model?


texts: How to label overlapping objects for deep learning model training

texts: I am training yolov3 to detect a custom object (chickens). In a lot of my training images I have overlapping chickens (can only see a partial chicken etc). Is there a common practice for how to label the data (bounding box) in these cases? Should you only label the portion of the image which you can see?


texts: Are application assignments in Datascience common practice?

texts:
this might seem off topic, but I hope this question will be tolerated.

I was recently asked to help a friend with an assignment in the process of applying for a new job as data scientist. And honestly I had a lot of fun solving the assignment for me and giving him guidance on how to proceed

However this got me wondering about a few things:

is it common practice in Datascience to hand out assignments to applicants?
is there a database available with datascience assignments, because I enjoy doing them?


texts: Could one algorithm fetch keywords from texts of different natural languages?

texts:
I am new to practicing NLP and most topics related, but I want to make a program that can gather and extract data for me on its own.

To be more specific, I want to tell the program "I want more information on this topic(i.e heart attacks)", and then the program shall find, gather and extract meaningful texts on the topic from around the www.

I happen to live in Norway, which means that most interesting data will be in English, but I also want to fetch interesting data found in Norwegian.

One challenge is the differences in stop words. For instance, "are" and "and" are both stop words in English and subjects in Norwegian.
Other challenges are also likely to occur.

So my question is: Would I need to create separate algorithms for every natural language to be interpreted?


texts: Why cant I overfit this dataset with my neural network?

texts:
I have read that given a model is complex enough and I train for enough epochs, my model should at some point overfit the dataset. However I implemented a simple neural network in keras and my validation set loss seems to never go up:
import tensorflow as tf
from tensorflow import keras
import numpy as np
import random
from sklearn import datasets, preprocessing
import matplotlib.pyplot as plt

# import and scale
dataset = datasets.load_boston()
X = dataset.data
y = dataset.target
X = preprocessing.scale(X)
y = y.reshape((y.shape[0], 1))

# shuffle
shuffle_indices = list(range(X.shape[0]))
random.shuffle(shuffle_indices)
X = X[shuffle_indices]
y = y[shuffle_indices]

# tain-validation split
X_train, y_train  = X[:int(X.shape[0] * 0.7)], y[:int(X.shape[0] * 0.7)]
X_val, y_val = X[int(X.shape[0] * 0.7):], y[int(X.shape[0] * 0.7):]

# define and fit model
model = keras.Sequential([keras.layers.Dense(X.shape[1], use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(y.shape[1])
                         ])
model.compile(optimizer=tf.keras.optimizers.SGD(
    learning_rate=0.0001
), loss='MeanSquaredError')

model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=500, verbose=1)

# plot data
plt.plot(range(1, len(model.history.history['loss']) + 1), model.history.history['loss'], label='Train Set Cost')
plt.plot(range(1, len(model.history.history['val_loss']) + 1), model.history.history['val_loss'], label='Validation Set Cost')
plt.xlabel("epoch")
plt.ylabel("loss")
plt.legend()
plt.show()

The model is a simple dense neural network with Mean Squared Error as its loss function and gradient descent as it's optimizer. I tried to make the network deeper, but the validation loss only keeps decreasing until it stops at one point.


texts: Make column as dictionary key and row as value in pandas dataframe

texts:
I have Dataframe as below.

name    value     id
A       123         1
B       345         5
C       567         4
D       789         2

I need to create a dictionary of the form below

{
  {"name":"A",
   "value":123,
   "id":1
  },
  {"name":"B",
   "value":345,
   "id":5
  },
  {"name":"C",
   "value":567,
   "id":4
  },
  {"name":"D",
   "value":789,
   "id":2
  }
}

I tried below code, but it makes 'name' as the key.

data_set.set_index('name').to_dict(orient='list')

How to get the dictionary of the form i have mention? Do i need to iterate through rows?


texts:
Cross-posted on Reddit ML.
Should a Feature Store be part of an enterprise data catalog?
To me, a feature store seems to be a highly niche data catalog but missing a lot of the benefits of having an enterprise data catalog / data discovery tool. My need is to have generated features discoverable when searching for data.
For example, if I have dataset A and B used to generate a feature set AB', I would want to know about that information if I search and ever come across dataset A or B in my data catalog.
Along with that, it would be beneficial to have the code / git commit that generated the features.
Am I missing something?


texts: How can generative models be used in machine learning classification applications?

texts: My understanding of generative models is that they generate data to match certain statistical properties. Intuitively, I find it hard how generative models can be used for classification purposes. On the other hand, discriminative models being used for classification is self-explanatory.


texts: How can I detect partially obscured objects using Python?

texts:
I'm building a computer vision application using Python (OpenCV, keras-retinanet, tensorflow) which requires detecting an object and then counting how many objects are behind that front object. So, objects are often overlapping.

For example: How many are in this queue?

Here, the aim is to detect the person in the front (foreground) of the queue, and then detect the amount of people behind the front person, despite those behind being obscured.

I have built an object recognition model using keras-retinanet to find the object in the foreground, but now I need to count how many are behind that object. My training data is thousands of images of objects ins straight lines, similar to the image linked above.

However, I have been unable to find a viable way to count the objects behind the front object when they are partially obscured (the rest of the people in the queue).

So far I have tried counting edges (using auto-thresholding), lowering the confidence threshold of my object detection model, and training a new object detection model using only obscured objects as training. These methods have all had very low accuracy.

Do you have any advice or direction on how I can attack this problem?


texts: Tensor Decomposition in TensorFlow for multinomial time series dimensionality reduction

texts:
I'm doing unsupervised learning (clustering and DR) on multinomial time series. I need to reduce dimensions for my data, which is sparse and has a lot of dimensions. I realized that some form of tensor decomposition may be the best way to go about this. What I have is a set of samples each with a set of features in time. Each time length can different, but there are the same features for each sample.

Want to end up with is a set of samples with a single feature in time. With that, I can do clustering using distance determined by dynamic time warping.

Does this entire scheme sound reasonable?

TensorFlow has a feature tf.qr (https://www.tensorflow.org/api_docs/python/tf/qr) which can do tensor decomposition, but I don't understand the output or math behind it well enough to utilize it. I have tried running it with what seems like success, but I don't understand what is stored in the q and r output tensors. If someone could explain this to me and if this even applies to what I am trying to do, I would really appreciate that.

In addition, I tried using scikit tensor's cp decomposition (https://github.com/mnick/scikit-tensor/blob/master/sktensor/cp.py) which  returns a single tensor. I also do not understand what is stored in this output or if it does something similar to TensorFlow's tf.qr function or related to what I want to do.

As you can probably tell by now, I am quite lost in the math. I've tried to find explanations of how these functions work, but I can't without jumping into multilinear algebra which I don't understand. If someone could point me in the right direction, I would greatly appreciate it.

EDIT:

if s1 and s2 are two multidimensional time series, then this should computer the distance between them. I tried it in python and it does in fact output a number, but I don't know if I am understanding this correctly. It behaves in the same way DTW would with one dimension but takes the sum of the MSE of all features instead of just one of them when computing the distance at a given point in time. Also, this DTW function has a time window which is set to 10 but of course can change...

def MDTWDistance(s1, s2, window=10, num_columns=1):

    DTW={}
    w = max(window, abs(len(s1)-len(s2)))

    for i in range(-1,len(s1)):
        for j in range(-1,len(s2)):
            DTW[(i, j)] = float('inf')
    DTW[(-1, -1)] = 0

    for i in range(len(s1)):
        for j in range(max(0, i-w), min(len(s2), i+w)):
            dist= mdist(s1.loc[i], s2.loc[j], num_columns)
            print i, j, dist
            DTW[(i, j)] = dist + min(DTW[(i-1, j)],DTW[(i, j-1)], DTW[(i-1, j-1)])

    return np.sqrt(DTW[len(s1)-1, len(s2)-1])

With...

def mdist(a, b, num_col):

    dist = 0
    for col in range(num_col):
        dist = dist + (a[col]-b[col])**2
        print a[col], b[col]
    return dist

Is this doing what I think it does/what the papers say is appropriate?


texts:
I am quite new to the concept of attention. I am working with graph data and running graph convolution on it to learn node level embedding first. Then an attention layer to aggregate the nodes to learn a graph level embedding. Here is the setup:

graph->Conv1(Filter size 128)->Conv2-(Filter size 64>Conv3(Filter size 32) -> Attention -> Some other layers

After three convolution pass i get a matrix of size number_of_nodes_in_the_graph X 32 (embedding length). After the attention layer i get a flat vector representation of the graph with length 32. Here is the forward function of the attention module:

    def forward(self, embedding):
        """
        Making a forward propagation pass to create a graph level representation.
        :param embedding: Result of the GCN.
        :return representation: A graph level representation vector.
        """
        global_context = torch.mean(torch.matmul(embedding, self.weight_matrix), dim=0)
        # print("Gloabal Context:", global_context.shape)

        transformed_global = torch.tanh(global_context)
        # print("transformed_global Context:", transformed_global.shape)

        sigmoid_scores = torch.sigmoid(torch.mm(embedding,transformed_global.view(-1,1)))
        # print("sigmoid_scores Context:", sigmoid_scores)

        representation = torch.mm(torch.t(embedding),sigmoid_scores)
        return representation, transformed_global

Now i would like to see which graph nodes were important for the final graph embedding. I am using pytorch for this. I cannot seem to figure out how to map the attention output to input. I would greatly appreciate any help! I am using this repo: https://github.com/benedekrozemberczki/SimGNN


texts: Synthetic Gradients good number of Layers & neurons

texts:
I would like to train my LSTM with a "synthetic gradients" Decoupled Neural Interface (DNI).

  How to decide on the number of layers and neurons for my DNI?
  Searching for them by trial end error or what's worse - by Genetic
  algorithm which would seem to outweigh [sic] defeat the purpose of Synthetic Gradients.

  And, if my DNI is an LSTM itself - it seems it would take even longer
  to determine its optimal structure

SG speed up the training speed, by allowing multiple forward passes (with immediate weight adjustments), since DNI will already predict the future gradient.

However, we will lose time "experiencing" a few hundreds of training sessions only to find a optimal structure of DNI with which it will predict gradient the best way.

By that time we could have already finished our training with an oldschool Backprop through Time.

  Also, how should we avoid our DNI overfitting, how to monitor and ensure it's not happening?


texts: What is the difference between data leakage and endogeneity?

texts:
I have the impression the former is used in ML whereas the latter is used in econometrics. They both carry the idea that information from the target is "leaking" in explanatory variables.

Is there any difference between those two notions?


texts: Classify Spanish Text into different Categories

texts:
I want to recommend articles to users depending upon what type of article is user reading, Music, Movies, Politics, etc.
I have 3 features: Page Title, Labels, article content.

I am using an API (meaning cloud) on the Page Title feature which follows a taxonomy(IAB) for categorizing the articles into different segments. But the API is not able to label everything. 50% Uncategorized.
I have tried using Basic ML models Naive Bayes for categorizing the remaining 50% by training the model on the categorized set, but the accuracy is low.

*What can be done to segment these articles into different clusters or segments so that a recommendation can be given for each type of article?
    from sklearn.svm import LinearSVC

    text_clf = Pipeline([('vect', CountVectorizer()),
                         ('tfidf', TfidfTransformer()),
                         ('clf', LinearSVC()),
                         ])

    text_clf.fit(X_train, y_train)

    predicted = text_clf.predict(X_test)

    print(metrics.classification_report(y_test, predicted))

                             Precision Recall   F1 Score    Support
              accuracy                           0.75       422
             macro avg       0.50      0.29      0.35       422
          weighted avg       0.74      0.75      0.72       422


texts: cosine similarity between items (purchase data) and normalisation

texts:
I'm using IndexedRowMatrix which represents the products's user purchase behaviours and in order to build product recommendations, I use cosine similarity to calculate similarities between products. PySpark provides a function called columnSimilarities() to do that.

My question is, do I need to normalize each product's vector before using columnSimilarities()? I read about normalisation and cosine similarity and understood that cosine similarity normalises vectors already, as if we normalize the vectors, then the cosine similarity will be just the dot product of the 2 vectors. Reference

Also, one of the answers in this question Cosine similarity versus dot product as distance metrics suggests that Sometimes it is desirable to ignore the magnitude, hence cosine similarity is nice, but if magnitude plays a role, dot product would be better as a similarity measure. which means cosine similarity and dot product are not the same..

I'm confused about the difference and when it's good to use normalisation before calculating cosine similarity and when it's not? and what are the different ways to normalize?

Any help?


texts: What will happen if almost constant values for features?

texts:
In a problem of an epidemiology dataset, is it desirable to keep the features that have almost constant values?
For example, In case of the feature, type_of_residence Large for 97 percent and Small for 2.7 percent of subjects. Is it okay to keep this feature?
My target variable is outcome of patients and this data set is unbalanced. Like oversampling and under sampling techniques in class imbalance problem, is there any process in ML to address other predicting features?
I am not doing feature selection using ML at this point for knowing the importance of this feature. But would like to know is there any general rules regarding keeping or not keeping these kind of features in dataset.


texts: Is it possible to predict sentiment of unlabelled dataset using BERT?

texts: I have a large unlabeled dataset and I want to predict sentiment for each document in this dataset. I want to know, is it possible that I can use BERT for sentiment analysis of unlabeled data? I have seen so many tutorials and read the blog posts but I couldn't find one. All shows the use of BERT on datasets that are already labeled such as the IMDB review dataset or Yelp review.


texts: How can discard "peaks" values from a chart?

texts:
I have this chart

where you can see that some points are far away from the "trend" of the curve. The chart represents $5,000$ values where each value corresponds to a specific hour of a day, month and year. In this case, it goes from  01/01/18 until 05/08/18 (non-american format) and as you can see at the first lower peak, it was caused because at 12.00 the cost was $20$ USD but at 1.00 AM it went up to $51$ USD.

I've read that some people just delete those 'peaks' values but I don't know if there's a better solution. This graph shows almost a year but I have in mind on graphing $10$ years of this prices so the situation could be worse. Do you know a better strategy? I'm thinking maybe in calculating average costs for every $6$ hours but it feels like a cheap solution to me.


texts: How does on test regression for a subspace or matrix factorization?

texts:
I've recently been reading a lot of papers and watching a lot of videos on both subspace learning, and matrix factorization.  One thing is particularly eluding me though - how does any of this get tested?

Let's take, straight from Wikipedia, Matrix Factorization Non-Negative.

$$V = WH$$

So, you have a data matrix $V$. Your goal is to learn components $W$ and $H$, which when multiplied together, give a good approximation of $V$. This can be done by minimizing over $W$, $H$

$$\| V - WH \|$$

That seems fine so far.  My problem, theoretically, is understanding when we want to apply this to a problem, like say Regression.

If you wanted to minimize:

$$Y - WH*B$$

How do you do this with a test point?  I get confused here, because if we had, say a 100-user test set with 10 features.  Then we do a 90/10 split, we get a size of $W*H$ that is different than the size of our test data.

Do people just plug the test data in directly when testing, in place of $W*H$, and just rely on those learned weights $B$?


texts: Regions /patterns in images that creates classification

texts:
I am  having  some images on which  I   trained some neural   network  models in order to  do classification( binary  task).
I am interested  of a  method    to identify  some patterns  or regions  in those images  that are  influencial  the classification   part in  other words  what  regions or  what patterns are formed  for a particular  class.


texts: Method for predicting winner of call for tenders

texts:
Introduction: Lately I've been looking into different machine learning methods to work around different business problems. By now I have a good, basic understanding of most regression and classification methods, and I'm able to use these methods to predict numeric values given other numeric values and/or simple categories (e.g. an employee's salary given age, years of experience and level of education) or a binary classification (e.g. will this employee leave the company based on the same variables).

What I'm looking for: However, I haven't found the right method for the problem I initially wanted to solve, which involves predicting a non-numeric, non-binary value from a mix of numeric and categorical data. I'm not looking for an in-depth explanation of how to solve the exact problem, but merely advise on which techniques/methods to look into. Ideally something that could be done with R.

The business problem: I have historical data on public tenders (i.e. public sector instutions buying goods/services from private contractors through calls for tenders). The data includes variables like:

Orderer - i.e. who announced the tender (1 of ~150 municipalities/state insitutions)
Type of procurement (1 or more of thousands of industrial classification codes)
Estimated value of contract - A numeric value estimating the value of the contract (at a point before the winner is chosen).
Winner - i.e. which contractor won the tender (1 of ~2000 private
companies)

What I want to do is predict the winner of a tender given the three other variables. It's obviously not a regression problem, and the classification methods I know seem inadequate in handling the problem too. The data are clean and streamlined (no alternate spelling of the different orderer/contractor names, etc.). Any ideas about what to look into?


texts: Improving SVM binary classification model on new dataset

texts:
I am working to create a SVM binary classifier for classification of Tweets based on news class "Crime" and "Non Crime". I have downloaded a dataset of 6400 rows from various sources and training my model on it. While I am able to achieve over 95% accuracy, what concerns me is about its performance on new datasets. The system will be deployed on live streaming tweets, so how can I put in a feedback system such as my model is continuously updates itself? As in, say for new crime type news source which were not covered in training model, how to incorporate them further?

I am asking a broad methodology question, not some specific program related question as I would like to figure out the implementation myself :)


texts: How to impute when you have multiple variables not important to the column with the missing values?

texts:
I have a dataset with 18000 rows and 192 columns. I have a specific column with more than 2000 rows missing.
I've tried some types of imputations but they just take to long or just seem not good enough. I wonder if it's okay to do some kind feature selection of the dataset to get the most important variables before doing a multivariate missing value imputation (like MICE). This way I could work with maybe 5 to 15 relevant variables for the imputation and then place the imputation on the original dataset.
Is there any problems to this aproach? like bias in the final dataset.
Thanks in advance


texts: LSTM: different number of rows at each time step

texts:
I have for each of the years N-k, ..., N a different data set. The data sets have the same number of columns but a different number of rows. The columns have the same meanings in the different datasets. One of the columns is the target.

The rows in the various datasets in no way correspond to each other. The 10th row in the 3rd dataset is not fundamentally connected to the 10th row in the 4th dataset - or any of the other rows. In future years' data sets, the rows will likewise be unconnected to the rows in the previous years' datasets.

Is LSTM even an option for solving this kind of problem?

Help appreciated.


texts: How is the "loss" calculated which is supplied by the callback log in Keras?

texts:
I.e. categorical cross entropy? binary cross entropy? Something else?

Or is it perhaps the loss function which you pass into the model.compile method?


texts: Statistical comparison of model performance when training and validation data is always the same

texts:
I have a labelled dataset of text in 4 languages (1000 samples per languages makes a total of 4000 samples). In one experiment I would like to assess the performance of a classification algorithm (neural network) on an unseen language. That means using the 3000 tweets from three languages as training data and 1000 samples from the fourth language as validation data:

training data:
- 1000 English samples
- 1000 German samples
- 1000 Italian samples

validation data:
- 1000 Dutch samples

Now, I am trying to find a statistic to find if there is a significant difference between 2 versions of the classification algorithm. I have been reading and find that 2x5 cross-validation or 10x10 cross validation using a modified student t-test is (one of) the best options. While I can apply this to most other experiments, I don't think I can apply this directly to this particular experiment since I am not using folds.

My question is, if I can still use the corrected student t-test training the network a number of times on the same training and validation data? If so, what should be the number of training and validation folds? If not, what would be a better approach?

For reference, below is my Python implementation of the Nadeau and Bengio correction using the equation stated here.

def corrected_dependent_ttest(data1, data2, n_training_folds, n_test_folds, alpha):
    n = len(data1)
    differences = [(data1[i]-data2[i]) for i in range(n)]
    sd = stdev(differences)
    divisor = 1 / n * sum(differences)
    test_training_ratio = n_test_folds / n_training_folds
    denominator = sqrt(1 / n + test_training_ratio) * sd
    t_stat = divisor / denominator
    # degrees of freedom
    df = n - 1
    # calculate the critical value
    cv = t.ppf(1.0 - alpha, df)
    # calculate the p-value
    p = (1.0 - t.cdf(abs(t_stat), df)) * 2.0
    # return everything
    return t_stat, df, cv, p


texts: How to predict multiple iterations in the future based on current close prices? Keras Python

texts:
I have data from CoinMarketCap for the close prices of Bitcoin. How can I use this data to predict multiple days into the future using past days with a function? I want the function to look like:

def predict(days, data):
  preds=data[-days*2:]
  for d in days:
     pred=model.predict(preds)
     preds.append(pred)
  return np.array(preds)

I already have a train-test split function and window converter.
My input data's shape is (*,1) and thus to loop this I need my output to have the exact shape.

How can I do this and what will my model look like in Keras? Do I have to convert the data into windows?

This is my code so far:

window_len=30

def train_test_split(df, test_size=0.1):
    split=len(df) - int(test_size * len(df))
    train=df[:split]
    test=df[split:]
    return train, test

def pred_convert_window(df,window_len=window_len):
    windows=[]
    for i in range(len(df)-window_len):
        windows.append(df[i:i+window_len])
    return np.array(windows)

# PRED MODEL IN/OUT
scaler=MinMaxScaler(feature_range=(0,1))
close=scaler.fit_transform(df['close'].values.reshape(-1,1))

train,test=train_test_split(close)

My close data shape is (2022,1)

My train and test data shapes are ((1820, 1), (202, 1))


texts: What is the best model for a recommendation system using implicit ratings?

texts:
I have a similariy matrix that looks like this:

I have a bunch of user vectors with 1s and 0s, with a 1 indicating that someone has clicked on an email (as part of a campaign) and zero to indicate they haven't. Implicit ratings as I have come to understand them.
In terms of an approach, after researching different options, it would appear that the best algorithm to start with a is a matrix-factorisation approach. My question is: what is the best algorithm to start as a baseline model for this type of problem? Any advice on potential pitfalls. Any help would be appreciated. Thanks!


texts: Reinforcement learning example when the action is a matrix

texts:
I am working on solving a problem with reinforcement learning which has to find the optimal matrix that maximize the reward. I am not able to see how I can formulate this problem as I have practiced example with simple actions like UP DOWN LEFT RIGHT. My actions are matrix of size (Nt, Nr) where Nt and Nr can be 64. I have tried to convert it to one dimension and use DDQN but when I run Q[action], I get an error because a list can only be indexed with an integer.
Please help me to find at least what I need to explore first. Do I need to use A2C networks?


texts: Machine learning with partially defined labelling using existential quantifiers

texts:
I apologize for a vague title, but I know only the basic machine learning jargon. Let's assume that we have domain set $X$ and label set $L = \left\{0, 1\right\}$, but the labelling function $f$ is only partially defined by function $g:P→L$, where:

(1)  $P$ is a partition of $X$

(2)  $\it g(p) = 0 ⟹ \forall x \in p: f(x) = 0$

(3)  $\it g(p) = 1 ⟹ \exists x \in p: f(x) = 1$

The meaning of function $g$ is that for some elements of $P$ all members have label $0$ and for remaining elements of $P$ at least one member has label $1$, but we don't know which is $1$ and which is $0$.

For example: we have boxes of apples with possibly different number of apples in each box. We know which boxes have all good apples (apple with label $0$) and remaining boxes have at least one bad apple (apple with label $1$), but we don't know which ones are bad or good.

Is there a machine learning technique to deal with such partially defined labelling?

EDIT

Let's assume that in boxes of apples example above, a classifier we looking for is: good apples are green and hard, bad apples are brown and soft. As a model of an apple, my lucky guess is a 4-d feature vector with color, hardness, weight and size components. All of the apples are observable, so we can calculate the feature vector for each of them. We know labels for all apples in "good" boxes, but have only an existential statement about labels of apples in "bad" boxes.

After measuring all apples, I look for clusters of feature vectors or run some unsupervised learning algorithm. I will discover 2 distinct classes: (green, hard, _, _) and (brown, soft, _, _). These classes will also satisfy conditions (2) and (3) above, so I will accept (green, hard) vs. (brown, soft) as a correct classifier.

With this procedure in mind, let me rephrase my question. Assuming my guess of feature vector is correct, so in about 10-d vector there are enough significant features for correct classification, the partition set $P$ has 1K elements and the domain set $X$ has 100K elements, what is the correct machine learning technique to use?


texts: What's the difference between ELM and NNAR?

texts: I'm working with time series forecasting using the two techniques that involve neural networks, the Extreme Learning Machine and the Autoregressive Neural Network. Reading the two methodologies, the first proposal by Huang et al. (2006) and the second explained in Dietz (2010) and Zhang (2003), cannot understand the main differences between the algorithms. Could more experienced people cite such differences?


texts: Why Orange "Predictions" and "Test & Score" produce different results on the same data?

texts:
It is not very clear what is the difference between the following two schemes:

From help docs:

Test & Score widget: tests learning algorithms on data.
Predictions widget: shows models’ predictions on the data. Which model? Pretrained one? Where is it taken from?

Obviously the results of Confusion Matrix are different. So how should I train and test my model is not clear.

If can easily do this:

What's the point of a separate single Predictions widget connected to some learner? What does it actually test?


texts: Best approach/model to match keywords to sentences

texts:
I'm very new to machine learning and AI stuff. I have the following problem I want to tackle and would like some advice as to how I should approach it:

Basically, I want to take some keywords and produce a sentence or list of sentences where the keywords are most relevant: []string -> []string.

The keywords could be any word, e.g. cat, dog, lemon, tea. and they would return a set of sentences that have been learned before.

So for example:

dog -> its a nice day
cat -> its not a nice day
lemon -> its a normal day
tea -> hello, world!

Would be a very basic data set. And then I would enter the keywords [ dog, tea ] and be given back [ "it's a nice day", "hello, world!" ]. In addition I would want to be able to train this model with new words and sentences.

At first glance I thought I could use a naive bayesian classifier? Though the model requires learning, and to my limited understanding of ML, I figure this would be some kind of neural network classifier.

Some research leads to me that NN are only capable of dealing with floating point/integer inputs? So I would have to map the strings somehow to this. Which is apparently called a "sequence to sequence" model.

Is this the way I would approach this problem? A sequence-to-sequence neural network? Or am I completely off the ball here.


texts: How can you estimate the value of a component part where you only know the value of a whole?

texts:
I'm not sure how to frame this question, or where to start. I'm new to data analytics, but looking to develop skills and knowledge.

An example of what I'm asking is if you have a retailers sales data where you have the total value of a given transaction, and details of each item related to a given transaction but don't have price data for each individual item, is it possible to estimate the value of each individual item given a large enough transaction data set?

The analogy breaks down a little for the use case I'm actually considering, as under this retailer example an item would likely have a fixed price. However, in my use case each item has a known arbitrary tariff value but an unknown actual value. We would only know an aggregated actual value in conjunction with other items which may be grouped under the same "transaction". Tariff and actual values will likely have a strong linear correlation with some variability, though this isn't known as actual values for individual items are not captured.

Hope this makes sense! Was wondering how define this problem, what approach would you take to a problem like this? And any links to related reading materials would be much appreciated.


texts:
In neural networks, the VC dimension $d_{VC}$ equals approximately the number of parameters (weights) of the network. The rule of thump for good generalization is then $N \geq 10 d_{VC} \approx 10 * (\text{number of weigts})$.
What is the VC dimension for Gaussian Process Regression ?
My domain is $X = \mathbb{R}^{25}$, meaning I have 25 features, and I want to determine the number of samples $N$ I must have to archive good generalization.


texts: Is it possible to find a model that minimises both false positive and false negative?

texts:
Is it possible to come up with a model that minimises both false positive and false negative?

Minimising can be done to a point, such as the Bayes error threshold.


texts: Non-Real Time Data Augmentation for CNN Classification. What are the drawbacks?

texts:
When people talk about and use data augmentation, are they mostly referring to real-time data augmentation? In the case of image classification, that would involve augmenting the data right before fitting the model, and a new augmented image is used every epoch. In this case only augmented images are used to train the model and the raw image is never used, so the size of the input doesn’t actually change.

But what about non-real-time data augmentation? By this, I mean augmenting the data in the preprocessing stage, so that you literally expand the sample size of your input. Then you feed all those augmented images along with the original into the cnn, so it’s the same images every epoch. Is this a valid idea? Has it been done and what are the drawbacks? Any logical fallacies or objections by data science and machine learning experts?

Thanks!


texts: Keras - Computing cosine similarity matrix of two 3D tensors

texts:
Using TF backend, I need to construct a similarity matrices of two 3D vectors, both with shape (batch_size, N, M), being N and M natural numbers.

The function tf.losses.cosine_distance is only between 1D tensors. I need to build a Tensor matrix batch_sizexNxM such that matrix[k][i][j] will be the cosine similarity of the Tensor1[k][i] and Tensor2[k][j].

How can I do this?


texts: Trying to use GPU of laptop for TensorFlow

texts:
I have an HP au111tx laptop with Nvidia 940mx graphics card.

Is it possible for me to use this graphics card exclusively for ML. I want to use the integrated graphics card for normal display. I can use Windows or Ubuntu.


texts: Should one hot vectors be scaled with numerical attributes

texts: In the case of having a combination of categorical and numerical Attributes, I usually convert the categorical attributes to one hot vectors. My question is do I leave those vectors as is and scale the numerical attributes through standardization/normalization, or should I scale the one hot vectors along with the numerical attributes?


texts: Concrete example to elaborate on the relationship between `linear function` and `aﬃne function` in the context of machine learning

texts:
I am learning the deeplearningbook

Section 5.1.4 says:

  so the mapping from parameters to predictions is still a linear
  function but the mapping from features to predictions is now an aﬃne
  function

Can any one give a concrete example to elaborate on the relationship between linear function and aﬃne function in the context of machine learning?


texts: Why non-differentiable regularization lead to setting coefficients to 0?

texts:
The L2 regularization lead to minimize the values in the vector parameter.
The L1 regularization lead to setting some coefficients to 0 in the vector parameter.

More generally, I've seen that non-differentiable regularization function lead to setting coefficients to 0 in the parameter vector. Why is that the case?


texts: What is the current best state of the art algorithm for graph embedding of directed weighted graphs for binary classification?

texts:
Sorry if this is a newbee question,  I'm not an expert in data science, so this is the problem:

We have a directed and weighted graph, which higher or lower weight values does not imply the importance of the edge (so preferably the embedding algorithm shouldn't consider higher weights as more important), they are just used to imply the timing of the events which connect the nodes, so the higher  weighted edges are events that have happened after the lower ones.

There can be multiple edges between nodes, and I want to do a binary classification using deep learning, meaning that my model gets a embedded graph vector as input and decides if its malicious(1) or not(0).

So what is the best state of the art graph embedding algorithm for this task that can capture as much information as possible from the graph? i read some graph embedding papers but couldn't find any good comparison of them since there are so many new ones.

IMPORTANT NOTE: One problem i have seen with some of the graph embedding algorithms, is that they try to have a small vector dimension, since i guess they are used in fields which there are a LOT of nodes so they need to do this, but in this task its not really important, the nodes are the functions in the program, and they very rarely go above 2000 functions, so even if the algorithm creates a 20k dimensions its no problem, I'm saying this because some of the algorithms that I'm reading will produce a vector that even has lower dimensions compared to number of the nodes in the graph! and that causes loss of information in my opinion. so to sum up, the performance and large vector size is not a problem in my task. so preferably the algorithm should gather as much  information  as possible from the graph.


texts: How to decide neural network architecture?

texts:
I was wondering how do we have to decide how many nodes in hidden layers, and how many hidden layers to put when we build a neural network architecture.

I understand the input and output layer depends on the training set that we have but how do we decide the hidden layer and the overall architecture in general?


texts: Multiple linear regression for multi-dimensional input and output?

texts:
Assume that I have $N$ points $x_i,i=1,...,N$ in some $A>1$-dimensional  space $\mathbb{R}^A$ with pointwise evaluations of some function $f:\mathbb{R}^A \rightarrow \mathbb{R}^B$, i.e. $f(x_i),i=1,...,N$ where $f(x_i) \in \mathbb{R}^B$.
It is my goal to find a multiple linear regression between $x_i$ and $f(x_i)$. Now sklearn has a function (sklearn.linear_model.LinearRegression) for a multiple linear regression for functions of the type $f:\mathbb{R}^A \rightarrow \mathbb{R} $, but my output is $B$-dimensional. I assume that I could make independent multiple linear regressions for each output dimension and then combine the results, but there must be an easier way of achieving this.
Do you know of a more efficient way?


texts: values passed to user-defined distance function by KNeighborsClassifier is wrong

texts:
I have a data-set in which all features are binary and the class of each data-point is also binary. I am trying to use KNearestClassifier with a user-defined distance function as follows:

KNN = KNeighborsClassifier(n_neighbors=3,
                           algorithm='ball_tree',
                           metric='pyfunc',
                           metric_params={"func": lev_metric})
x_train, x_test, y_train, y_test = train_test_split(df_sum,
                                                    y,
                                                    test_size=0.1,
                                                    random_state=0)
KNN.fit(x_train, y_train)

and my custom metric function is as follows:

def lev_metric(a, b):
    print(a)
    print(b)
    return levenshtein(a, b)

the metric function expects two ndarrays of binary values of 0s and 1s. When knn.fit calls the metric function, "b" looks as expected (e.g. [0 1 1 0 0 1 0 1...) but "a" looks like gibberish and is an ndarray with real valued elements between 0 and 1, for example :

[0.32222222 0.42222222 0.34444444 0.47777778 0.41111111 0.38888889
 0.4        0.31111111 0.35555556 0.35555556 0.42222222 0.46666667
 0.36666667 0.32222222 0.41111111 0.32222222 0.36666667 0.35555556
 0.41111111 0.33333333 0.4        0.42222222 0.3        0.37777778
 0.38888889 0.48888889 0.41111111 0.43333333 0.34444444 0.35555556
 0.43333333 0.38888889 0.43333333 0.32222222 0.47777778 0.34444444...

what am I missing? I have also checked that the "x_train" is correct. Also, isn't knn an instance based learner? why is it calling the distance function in fit anyways? is it not supposed to just memorise the training examples? Thanks.


texts: lookup and fill some value from one dataframe to another

texts:
I have 2 dataframes, df1,and df2 as below.
df1

and df2

I would like to lookup "result" from df1 and fill into df2 by "Mode" as below format. Note "Mode" has become my column names and the results have been filled into corresponding columns.
also note that "ID" from df2 may not necessary equal to "ID" from df1.For example, I am only interested in 4 IDs (A01,A03,A04 and A05, no A02)
while df1 may contain more IDs

I tried to use below code but it doesn't give me a good result.
merged_df = pd.merge(df2, df1,left_on = 'ID', right_on = 'ID', how='outer')
How can I do this?


texts: Learning to rank: how is the label calculated?

texts:
I am studying learning to rank and not sure I understand how the train sample and final label (relevance score) is constructed.

Lets assume we sell furniture online. We have logged customer's query, product customer bought, clicked.

Example:
User A searched for "red sofa", clicked on Pa(r=1), Pb(r=5), Pc(r=10) and bought Pc.
User B searched for "red sofa", clicked on Pa(r=1), Pd(r=2), Pe(r=6) and bought Pd.
User C searched for "blue chair", clicked on Pu(r=1), Po(r=2), Ps(r=6) and bought Pu.

(r stands for absolute position from top to bottom of a product).
(lets say for clicking we would add 1, for buying we would add 2)

Case 1:
Without any personalization, I just want to rank the products only based on textual data from the product and query. How do I construct my training data?

Two users searched for "red sofa", do I treat those separate, e.g per user per query?

user     query         product     relevance  some_features
A        "red sofa"    Pc             2
A        "red sofa"    Pa             1
A        "red sofa"    Pb             1
A        "red sofa"    other products 0
B        "red sofa"    Pd             2
B        "red sofa"    Pa             1
B        "red sofa"    Pe             1
B        "red sofa"    other products 0
C        "blue chair"  Pu             2
C        "blue chair"  Po             1
C        "blue chair"  Ps             1
C        "blue chair"  other products 0

This looks to me a bit weird because how should an algorithm understand how products Pc or Pd (bought by user A and user B) should be ranked compared to each other?

Or do I treat logs for the same query together, e.g aggregating per query? (Because user A and B clicked both on Pa, it got a relevance of 2=1+1)

query         product     relevance  some_features
"red sofa"    Pc             2
"red sofa"    Pa             2
"red sofa"    Pb             1
"red sofa"    Pd             2
"red sofa"    Pe             1
"red sofa"    other products 0
"blue chair"  Pu             2
"blue chair"  Po             1
"blue chair"  Ps             1
"blue chair"  other products 0

This also looks weird to me because if users click the same products but buy different ones, the relevance of clicked products will be higher.

Question 1: How should I represent a label and a training sample in this case?

Case 2:
With personalization. If I want to provide user's features like "clicked before", "put into favorites", I would indeed construct a training record per user, query.

Question 2: Is it correct to do so?


texts:
I have a very interesting question. It's one month ago Geoffrey Hinton release his CapsNet paper. It's a complete new way for visual computing in the neural network stack. But my question is. Is it possible to use the CapsNet as a GAN.
I only understand the concept of the CapsNet but not technical limitations. So could this be possible?

This sounds for me for a nice field to research:)


texts: Multiple XGBoost models or just 1 for a cetain type of category?

texts:
I am building a model to predict, say house prices. Within my data I have sales and rentals. The Y variable is the price of either the sales or rentals. I also have a number of X variables to predict Y, such as number of bedrooms, bathrooms, meters squared etc.

I believe that the model will firstly make a split on the variable "sales" vs "rentals" as this would reduce the loss function - RMSE - the most.

Do you think it is best to train 2 models one for "sales" and the other for "rentals"? The RMSE for the model is quite high and this is in part due to the incorrect "Sales" predictions.


texts: Building document classifier based on keywords, what would be the steps?

texts:
I have a requirement of classifying documents(.doc files) based on the profiles. I have a csv file with data:

label          keyWords
Web developer  ["html","css","php"]
Developer      ["core","java","python"]
Embedded Dev   ["ARM", "CORTEX", "C"]

Now I want to classify the .doc files.

Also please do tell me what model to use ? or broad steps for proceeding (like building dataframe-> then splitting data and so on...)

EDIT:Any pointers on how to proceed with document classifier ?


texts:
I am looking for two types of data

Demographic data
Is there a vendor or data source that could provide demographic information of customers who buy a product? It doesn't include personal information but might have information like which zipcode the customer live, age etc..
Sentiment data
In addition to social media and e-commerce, are there any vendors sell sentiment data for products, brand etc.?


texts: How to calculate the silhouette coefficient?

texts:
Calculate the silhouette coefficient of point Pi from the above image.

To apply the given formula, how to know which is a(i) and b(i)?


texts: Consequences of using XGBoost regressor for small dataset(< 500 rows)

texts:
I am using XGBoost regressor to train my model for 322 rows of data and the train and test split is as follows: ((257, 9), (257,), (65, 9), (65,))
I am using the following parameters for hyper-parameter tuning:
{'max_depth': 3,
 'min_child_weight': 6,
 'eta': 0.3,
 'subsample': 0.9,
 'colsample_bytree': 0.7,
 'objective': 'reg:linear',
 'eval_metric': 'rmse',
 'reg_lambda': 0,
 'reg_alpha': 0.5,
 'gamma': 0}

I am getting the following results:
Train results:

MAE =  43.95317769328908
RMSE =  69.32233101307436
R2 score =  0.7500463354991436

--------------------------------------------

Test results :

MAE =  51.21307032658503
RMSE =  79.65759750390318
R2 score =  0.6569142423871053

What are the drawbacks of training XGBoost model on such a small dataset? I know about overfitting, but I can control it to some extend with regularization.


texts: Question Regarding Multi-label probability predictions

texts:
I have been doing a problem in which I have to predict probabilities for each of the labels in a multi-label (four to be precise) classification problem.

Example of a solution:

Id, North,          East,            West,           South
1,  0.71663940211,  0.037567315693,  0.03525987339,  0.0021068944991
...

The training data is of the form where each y(i) is labelled either 0,1,2 or 3 (encoded for N,E,W & S respectively)

I will be grateful if you can just tell me how to approach this problem.
Links giving direct insight to the problem will also sufficient.


texts: Model trained on CIFAR 10 performing badly when fed with internet downloaded images

texts:
My model is based on Shallow Net.
When I am training my model, the results are:
loss: 1.1398 - accuracy: 0.6093 - val_loss: 1.2309 - val_accuracy: 0.5657

Then I downloaded 20 images (2 for each class) from the net to check the performance.

Labels corresponding to this dataset should be:0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9].
But my model's prediction is: [0,0,0,1,0,1,5,0,0,5,2,2,0,0,5,2,0,0,1,9].
The accuracy is: 0.2 which is quite low as compared to 0.5657.
My code to load these datasets:
for file in os.listdir("C:/Users/....."):
    img_arr=cv2.imread(os.path.join(os.getcwd(),"Dataset",file))
    img_arr=cv2.resize(img_arr,(32,32))/255
    img_arrs.append(img_arr)

img_arrs=np.array(img_arrs)
img_arrs=img_arrs.reshape(20,32,32,3)
model=load_model("weights.hdf5")
pred=model.predict(img_arrs).argmax(axis=1)

What could be the reason behind this? Can someone give me an insight?
Edit:(Added training code)
(x_train,y_train),(x_test,y_test)=cifar10.load_data()
x_train=x_train.astype(float)/255
x_test=x_test.astype(float)/255
lb=LabelBinarizer()
y_train=lb.fit_transform(y_train)
y_test=lb.transform(y_test)

labelNames = ["airplane", "automobile", "bird", "cat", "deer","dog", "frog", "horse", "ship", "truck"]

model=ShallowNet.ShallowNet.build(width=32, height=32, depth=3, classes=10)
sgd=SGD(0.001)
model.compile(optimizer=sgd,loss="categorical_crossentropy",metrics=["accuracy"])
H=model.fit(x_train,y_train,validation_data=(x_test,y_test),batch_size=32,epochs=50,verbose=1)


texts: How to use ColumnTransformer and FunctionTransformer to apply the same function to many columns, but separately?

texts:
I want to apply pd.cut as a transformer in a pipeline, like this:

numerical_preprocessing = Pipeline([
            ('cut_into_bins', FunctionTransformer(pd.cut, kw_args={'bins': [10, 100, 1000]})
             )]

However, I get an error: ValueError("Input array must be 1 dimensional")
I could just write the same function over and over again with each column separately, but that looks like a terrible coding practice. Any thoughs on this?


texts:
I have noticed that when you make a small decision tree model, and then extend the model by creating an ensemble of trees around the same tree settings, the variable importance is diluted in the sense that the least important and most important variables become a lot more closer together. In some cases, there may be almost do distinction in importance.
Are there methods available to either mitigate this effect or to measure it, with a view to defining any tradeoff between understanding variable importance and overall accuracy?


texts: Training is not stable with extreme class imbalance

texts:
I'm dealing with a multi-class classification problem with around 30 categories.
This problem has a severe class imbalance:

Around 300 examples for the least common class.
Around 100k examples for the most common class.

I don't want the classification model to be dummy and predict the most common class for most of the examples, for this reason, I'm using class_weight='balanced' in my LogisticRegression from sklearn. However, in this case, the classes that the algorithm predicts are mostly the less frequent ones. I understand the model overfits them somehow, as it assigns every sample from these class a very high weight.
On the other hand, if I don't apply the class weights, the model predicts the most common categories.
Is there a way to solve this? Is there a way to ensure the model predicts approximately the same proportion of samples for each category?


texts: When does it makes senses to use Dot-Product as similarity measure instead of Cosine?. I have seen there is already question asked about this, However that merely explains the difference between calculation of dot-product & cosine and it does not focus on when should we use one vs another with real world example.


texts: Is the role of the validation set in a deep learning network is only for Early Stopping?

texts:
In the "deep learning crash course" given by Leo Isikdogan in lecture 4 https://www.youtube.com/watch?v=ms-Ooh9mjiE&list=PLWKotBjTDoLj3rXBL-nEIPRN9V3a9Cx07&index=4
Overfitting, Underfitting, and Model Capacity
, He suggests that the data should be split in train, validation and test sets. The Train set is used to train the model, the validation set to optimize the hyperparameters and the test set to give an unbiased estimation of the generalization error.
When I look at how people implement the design, they usually use gridseachCV to evaluate the deep learning neural network to configure some of the hyperparameters such as the number of neurons, learning rate, optimizer, etc. After that, they used the validation set to perform the early stopping.

I think that it is possible to obtain a hyperparameter configuration from gridsearchCV that gave the best performance because it is already overfitted. Is it correct?

Is there another option to tune the hyperparameters with a validation set. Why they do not try different configurations in the validation set?


texts: Handling categorical variables in linear regression and random forest

texts:
In the linear regression, when we have a categorical explanatory variable with $n$ levels, we usually remove one level and call it a baseline level and fit the model on the remaining levels. And the final intercept is the intercept plus the coefficient of baseline level. Now my questions are:

Does it matter which level I choose to remove? I am working on a dataset to predict house price. When I use linear model with intercept on the testset for prediction, I get negative values for some houses which is not correct, as house price cannot be negative. But when I fit a regression without an intercept all the prediction values are positive as expected. I guess it has to do something with the baseline level I chose.
Do I have to remove one level from all my categorical variables in the dataset before fit a random forest, KNN, Ridge, Lasso,... too?!


texts: How to run Spark python code in Jupyter Notebook via command prompt

texts:
I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below

After that I want to run this in CMD so that I can save my python codes in text file and save as test.py (as python file). Then, I run that python file in CMD using python test.py command, below the screen shot:

So my task previously worked, but after 3 or 4 hours later I again tried to execute the same process, with no changes made...Then unfortunately I got the following error message:
Can someone please explain why this has happened?
Many thanks in advance


texts: How can I predict call traffic for the telecom company

texts: I have to predict the call traffic for a telecom company.